JP2007164383A - Marking system for photographing object - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a marking system for a photographing object which can quickly and easily calculate position of the object on a photographed image when presenting information relating to the object on the photographed object of a mobile terminal. <P>SOLUTION: The marking system for photographing object comprises an image inputting means 102, an object designating means 103, a marker coordinate measuring means 105 for measuring the coordinate of the marker on the photographed image, a marker information storing means 104 for storing three dimensional coordinate of the object, a coordinate transforming parameter calculating means 106 for obtaining a coordinate transforming equation to calculate the coordinates which is made by projecting the three dimensional coordinates on the photographed image by using a three dimensional coordinates of a plurality of markers and the coordinates on the photographed image, a coordinate calculating means 107 for calculating the coordinate of the object on the photographed image, and an image outputting means 108 for plotting the position of the object. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a photographing object labeling system that searches for exhibits in exhibitions, museums, and the like, and supports guidance for sightseeing spots and theme parks.

  In recent years, digital cameras, mobile phones with digital cameras, electronic notebooks with digital cameras, etc. have been used for mobile terminals with photographing means, such as searching for exhibits at exhibitions and museums, and supporting tourist sites and theme parks. A photographing object labeling system to be performed is implemented, and many inventions for displaying guidance information related to the image on an image photographed by photographing means have been made as guidance assistance.

  As a portable terminal having such a conventional photographing means (see, for example, Patent Document 1), based on the image photographed by the photographing means, the position and direction of the photographing means are calculated to identify the subject, Some display information related to the subject on a captured image.

As another conventional technique (for example, see Patent Document 2), a plurality of markers whose three-dimensional positional relationship with the subject is known are photographed, and the positional relationship with the subject marker is based on the photographed image. There is one that calculates the three-dimensional posture position of the subject that is the object to be photographed, and a method of calculating and calculating a conversion matrix for converting the camera coordinate system and the world coordinate system at the time of this position calculation. is there.
Japanese Patent Laid-Open No. 2002-92012 JP 2000-227309 A

  However, in the conventional imaging object labeling system as described above, in order to calculate the position of the subject on the captured image, the position of the imaging means and the direction of the optical axis of the imaging means are measured. There was a need to do.

  In addition, in order to calculate the position of the subject using a plurality of markers, the coordinates of the subject and the coordinates of the marker must be managed. There is a problem that the coordinate measurement of the subject and the marker has to be repeated every time it moves.

  The present invention solves the above-described conventional problems, and when displaying information related to a subject on a captured image in a portable terminal having a photographing unit, the position of the subject on the captured image is reduced for a short time. It is possible to provide a photographing object labeling system that can be easily and accurately calculated.

  In order to solve the above-described problem, the imaging object labeling system according to claim 1 of the present invention calculates and outputs coordinates on the captured image of the subject using a plurality of markers associated with the subject. An image input unit for inputting the captured image, a subject designating unit for designating the subject using the unique identification information of the subject, and a subject to be designated by the subject designating unit And sign information storage means for storing the three-dimensional coordinates of the plurality of markers, marker coordinate measurement means for measuring the two-dimensional coordinates of the plurality of markers taken on the photographed image, and a plurality of photographs taken on the photographed image Using a three-dimensional coordinate of a marker and a two-dimensional coordinate on the photographed image, a coordinate conversion formula for calculating a two-dimensional coordinate obtained by projecting an arbitrary three-dimensional coordinate onto the photographed image Coordinate transformation parameter computing means for computing parameters, coordinate computing means for computing the two-dimensional coordinates of the subject projected on the photographed image from the three-dimensional coordinates of the subject using the coordinate transformation formula, And an image output means for drawing the position of the subject on the photographed image.

  The imaging object labeling system according to claim 2 of the present invention is the imaging object labeling system according to claim 1, wherein a coordinate change detection means for detecting a change in three-dimensional coordinates of the subject, and the label And a three-dimensional coordinate updating unit that updates the three-dimensional coordinates of the subject stored in the information storage unit in accordance with a detection result of the coordinate change detection unit.

  The imaging object labeling system according to claim 3 of the present invention is the imaging object labeling system according to claim 1 or 2, wherein the object is represented by a combination of a plurality of three-dimensional coordinates. Subject range output means for indicating a range on a photographed image is provided.

  Moreover, the imaging | photography target object labeling system of Claim 4 of this invention is the imaging | photography object labeling system of Claims 1-2, Comprising: The said object was shown by showing the said subject with the combination of several three-dimensional coordinates. Subject range masking means for concealing the output range of the subject on the captured image from the captured image by image processing is provided.

  In addition, the imaging object labeling system according to claim 5 of the present invention is the imaging object labeling system according to claim 1 or 2, wherein the object is represented by a combination of a plurality of three-dimensional coordinates. Subject image insertion means for replacing an output range on a photographed image of a subject with another image is provided.

  According to a sixth aspect of the present invention, there is provided a photographing object labeling system on a photographed image of a subject using a plurality of markers associated with the subject and having direction indicating means for indicating a three-dimensional direction to the subject. An imaging object labeling system that calculates and outputs the coordinates of the image, wherein the image input means inputs the captured image, the subject designation means designates the subject using the unique identification information of the subject, and the subject designation Marking information storage means for storing a subject to be specified by means and three-dimensional coordinates of the plurality of markers, marker coordinate measuring means for measuring two-dimensional coordinates of the plurality of markers shown on the photographed image, and the photographing Marker direction measuring means for measuring a direction indicated on the captured image by a plurality of marker direction indicating means on the image, and an association with the subject on the captured image Coordinate calculation means for calculating, as the two-dimensional coordinates of the subject, the two-dimensional coordinates of the intersection of the straight lines passing through each of the plurality of markers and directed in the direction indicated by the direction indicating means of each marker, Image output means for drawing the position of the subject on the photographed image.

  According to a seventh aspect of the present invention, there is provided a photographing target marking system according to the sixth aspect, comprising: a subject movement detection means for detecting movement of the subject; and a subject movement detection means. And marker direction instruction information updating means for updating the direction instruction means for the marker based on a detection result.

  According to an eighth aspect of the present invention, there is provided a photographing target marking system according to the first to seventh aspects, wherein the photographing information marking system stores guide information of the subject, and the captured image. And guide output means for displaying the guide information of the subject in the guide information storage means in association with the above-described two-dimensional coordinates of the subject.

  Moreover, the photographing object labeling system according to claim 9 of the present invention is the photographing object labeling system according to claims 1 to 8, wherein the lens distortion of the photographed image input from the image input means is corrected. Image correction means for outputting a corrected image is provided.

  The imaging object labeling system according to claim 10 of the present invention is the imaging object labeling system according to any of claims 1 to 9, and a peripheral image input unit that captures and inputs a peripheral image around the imaging range. Image synthesizing means for creating a composite image by combining the photographed image input from the image input means and the peripheral image from the peripheral image input means; and composite image input means for inputting the composite image from the image synthesizing means It is characterized by comprising.

  The imaging object labeling system according to claim 11 of the present invention is the imaging object labeling system according to claims 1 to 10, wherein the marker appears on the captured image input from the image input means. A marker concealing means is concealed from the photographed image by image processing.

  The imaging object labeling system according to claim 12 of the present invention is the imaging object labeling system according to any of claims 1 to 11, wherein the marker recognition system stores a marker recognition rule recognized by the marker coordinate measuring means. A rule storing means is provided.

  As described above, according to the first aspect of the present invention, in the photographing object labeling system that indicates the position of the subject on the photographed image using the plurality of markers that are placed in the photographing space and displays the unique identification information, By calculating the projection matrix element of the coordinate transformation from the two-dimensional coordinates on the photographed image and the three-dimensional coordinates in the real space of the marker, the subject's three-dimensional coordinates can be calculated from the subject's three-dimensional coordinates without calculating the position and direction of the photographing means. Since it is possible to construct a system for obtaining two-dimensional coordinates on a photographed image, it is possible to construct a system that can be easily searched while photographing an object that cannot be directly marked, such as a cultural property or a distant view.

  According to the second aspect of the present invention, the subject coordinates stored in the sign information storage means are updated as the subject coordinates change, so that the subject position on the photographed image can be changed even if the subject position varies with time. Can be constructed. In addition, by storing the coordinates of the subject at a specific time in the sign information storage means in advance, the subject's past and future coordinates can be displayed on the photographed image, for example, by expressing the movement of the subject as a vector display on the photographed image. It is possible to build a system that can

  According to the third aspect of the present invention, for example, a system that can indicate a range occupied by a subject in a captured image can be constructed by associating a point indicating a solid shape including the subject with the subject.

  According to the invention of claim 4, the subject on the photographed image is concealed by performing image processing such as a mosaic so that a subject that is in danger of privacy infringement or copyrighted material is photographed. A system that can be excluded from the image can be constructed.

  According to the invention of claim 5, it is possible to construct a system capable of supplying services such as insertion of an advertisement image at the time of photographing or image browsing by replacing the subject on the photographed image with another image. it can.

  According to the invention of claim 6, when the influence of aberration such as lens distortion is small on the photographed image, the projection property that the straight line passing through the subject in the three-dimensional space passes through the subject even if projected on the photographed image. By calculating the intersection of two or more straight lines passing through the subject, the position of the subject on the captured image can be easily determined without measuring the coordinates of the subject and the marker that guides the position of the subject. It is possible to construct a system that can show

  According to the invention of claim 7, it is possible to construct a system that obtains the same effect as that of the invention of claim 2 by updating the direction information indicated by the direction indicating means for each marker as the subject moves. it can. In addition, by generating a signal that emits light before and after the direction indicated by the marker direction instruction means is updated and adding it to the input, the number of markers to be remeasured is reduced, the response speed of the system is improved, Disturbance of direction measurement during information update can be prevented.

According to the eighth aspect of the invention, it is possible to construct a system that displays guide information such as the name and description of the subject in accordance with the position of the subject and the portion of the subject.
According to the ninth aspect of the present invention, when a captured image having a large influence of aberration captured by a wide-angle lens or the like is input, by having a corrected image input means for inputting a corrected image in which the aberration is corrected, A system that can improve the accuracy of the calculation of the position of the subject on the image can be constructed.

  According to the invention of claim 10, even when the number of markers for guiding the position of the subject around the subject is small, such as during magnified photographing, the position of the subject can be determined by including the marker around the photographed image in the measurement. It is possible to construct a system that shows accuracy.

According to the eleventh aspect of the present invention, it is possible to construct a system that makes it easy for the photographer to see the photographed image by hiding the marker on the photographed image from the photographed image.
According to the twelfth aspect of the present invention, it is possible to construct a system capable of selecting a marker according to a shooting environment such as an indoor or outdoor lighting condition or an arrangement of a shooting target object by switching a marker recognition rule. it can. In addition, it is possible to construct a system that uses an object within a shooting range that can be used at a shooting location in a specific environment such as an office lighting arrangement or a road sign as a marker.

  As described above, the position of the subject on the captured image can be easily and accurately calculated in a short time when displaying the information related to the subject on the captured image of the portable terminal having the photographing means.

Hereinafter, a photographing object labeling system showing an embodiment of the present invention will be specifically described with reference to the drawings.
(Embodiment 1)
An imaging object labeling system according to Embodiment 1 of the present invention will be described.

  FIG. 1 is a configuration diagram of a photographing object labeling system according to the first embodiment. The imaging object labeling system according to the first embodiment is an imaging object labeling system that indicates a position of a subject on a captured image using a plurality of markers that are placed in an imaging space and display unique identification information. As shown in FIG. 1, an image input unit 102 for a user 101 to input a captured image, a subject specification unit 103 for the user 101 to specify a subject using the unique identification information of the subject, and a subject to be specified by the subject specification unit 103. A sign information storage unit 104 that stores a subject and three-dimensional coordinates of the plurality of markers associated with the subject, a marker coordinate measurement unit 105 that measures the two-dimensional coordinates of the plurality of markers reflected on the photographed image, and the photographing 2 on the photographed image is obtained from arbitrary three-dimensional coordinates using the three-dimensional coordinates of a plurality of markers appearing on the image and the two-dimensional coordinates on the photographed image. Coordinate conversion parameter calculation means 106 for calculating the parameters of the coordinate conversion formula for calculating the original coordinates, and the two-dimensional coordinates of the subject projected on the captured image from the three-dimensional coordinates of the subject using the coordinate conversion formula. A coordinate calculation means 107 for calculating, and an image output means 108 for drawing the position of the subject on the photographed image on the photographed image and outputting it to the user 101 are provided.

  In FIG. 1, a user 101 photographs a subject and a marker arranged based on the three-dimensional coordinates of the subject and marker stored in the sign information storage unit 104, and inputs the captured image to the image input unit 102. . FIG. 2 shows an example of a photographed image 200 input by the user 101 to the image input unit 102. Each marker 201 has unique identification information, and the unique identification information is distinguished on the captured image 200 by means such as color, shape, barcode, and light emission pattern.

  The image input unit 102 sends the captured image 200 input from the user 101 to the marker coordinate measurement unit 105. The subject specifying unit 103 outputs the unique identification information of the subject stored in the sign information storage unit 104 to the user 101, and the unique identification of the subject as a target indicating the position on the captured image 200 selected and input by the user 101. Information is sent to the coordinate calculation means 107. FIG. 3 shows the unique identification information of the subject stored in the sign information storage unit 104 in a table 301 superimposed on the photographed image 200 and the subject designation unit 103 indicates to the user 101. The user 101 selects “product A”. It is an example of the output image 300. FIG. It should be noted that the user 101 can set in advance the display of all subjects and the types of subjects to be displayed such as road guide signs and merchandise as a prescribed selection.

  The sign storage unit 104 holds the unique identification information and three-dimensional coordinate information of the subject and the marker. FIG. 4 is an example of subject and marker unique identification information and three-dimensional coordinate information stored in the sign storage means 104. In the example of FIG. 4, the subject and the marker are distinguished from each other as a marker when the name is used as unique identification information and “marker” is at the head of the name. The marker coordinate measuring means 105 recognizes the marker 201 on the photographed image 200 by image processing, measures the unique identification information of each marker and the two-dimensional coordinates on the photographed image 200, and sends them to the coordinate conversion parameter calculation means 106.

  The coordinate conversion parameter calculation unit 106 searches the marker information storage unit 104 for the three-dimensional coordinates of the marker measured by the marker coordinate measurement unit 105 using the unique identification information of the marker. Together with the two-dimensional coordinates, a conversion matrix for conversion from the world coordinate system to coordinates on the photographed image 200 is obtained, and parameters of the obtained conversion matrix are sent to the coordinate calculation means 107.

  The coordinate calculation means 107 performs matrix calculation on the three-dimensional coordinates sent from the subject designation means 103 based on the parameters sent from the coordinate conversion parameter calculation means 106, and the obtained two-dimensional coordinates and the subject designation means 103 The sent unique identification information is sent to the image output means 108. The image output means 108 outputs the unique identification information and two-dimensional coordinates sent from the coordinate calculation means 107 to the user. FIG. 5 shows an example of an output image 500 in which “product A” 501 is displayed in the form of a balloon at the coordinates of the product A 202 on the photographed image 200 as the operation of the image output unit 108 of the first embodiment.

  As described above, according to the first embodiment, the position and direction of the photographing means are obtained by obtaining the projection matrix elements of the coordinate transformation from the two-dimensional coordinates on the photographed image of the marker and the three-dimensional coordinates in the real space of the marker. It is possible to construct a system for obtaining the two-dimensional coordinates on the photographed image of the subject from the three-dimensional coordinates of the subject without calculating.

  In the first embodiment, the sign information storage means 104 is mounted in the photographing object sign system. However, the sign information storage means 104 prepares temporary storage means such as wireless or wired communication or downloading in advance before using the system. Thus, the sign information storage means 104 can be arranged near the subject or the marker or on the Internet. Further, by recording the information stored in the sign information storage means 104 and the measurement result of this system in a data area other than the image of the photographed image or a file written in a markup language such as XML or HTML, Link information such as hyperlinks can be attached to a subject in a captured image when the captured image is played back by a network browser or the like.

Further, in the first embodiment, the marker identification information and the three-dimensional coordinates are stored in the marker information sign storage 104, but the marker identification information includes the marker three-dimensional coordinate information or the unique information including the three-dimensional coordinate information. By using the identification information, the marker coordinate measuring means 105 can acquire the three-dimensional coordinate information of the marker when measuring the unique identification information of the marker.
(Embodiment 2)
A photographing object labeling system according to Embodiment 2 of the present invention will be described.

  FIG. 6 is a configuration diagram of the photographing object labeling system according to the second embodiment. As shown in FIG. 6, the imaging object labeling system according to the second embodiment includes the components according to the first embodiment, coordinate change detection means 609 that detects a change in the three-dimensional coordinates of the subject, and sign information storage means. A three-dimensional coordinate update unit 610 updates the three-dimensional coordinates of the subject stored in 604 according to the detection result of the coordinate change detection unit 609. In the figure, components 601 to 608 correspond to components 101 to 108 of the first embodiment shown in FIG.

  In FIG. 6, a coordinate change detection unit 609 detects the movement and updated three-dimensional coordinates of the subject by a camera, a sensor, or the like that individually monitors the subject, and updates the subject-specific identification information and updates to the three-dimensional coordinate update unit 610. Send the generated 3D coordinates. The three-dimensional coordinate update unit 610 updates the information stored in the sign information storage unit 604 with the unique identification information sent by the coordinate change detection unit 609 and the updated three-dimensional coordinate by wireless communication means. .

As described above, according to the second embodiment, by updating the subject coordinates stored in the sign information storage unit 604 in accordance with the change in the coordinates of the subject, the photographed image can be obtained even if the position of the subject varies with time. A system that can display the position of the subject can be constructed.
(Embodiment 3)
A photographing object labeling system according to Embodiment 3 of the present invention will be described.

  FIG. 7 is a configuration diagram of the photographing object labeling system according to the third embodiment. As shown in FIG. 7, the imaging object labeling system according to the third embodiment represents a subject with a combination of the components of the first embodiment and a plurality of three-dimensional coordinates. It comprises subject range output means 709 indicating the range. In the figure, components 701 to 708 respectively correspond to components 101 to 108 of the first embodiment shown in FIG.

  The difference in operation from the first embodiment is that the coordinate calculation means 707 performs matrix calculation on the plurality of three-dimensional coordinates sent from the subject designation means 703 based on the parameters sent from the coordinate conversion parameter calculation means 706. Then, the obtained plural two-dimensional coordinates and the unique identification information sent from the subject specifying means 703 are sent to the subject range output means 709.

  The subject range output unit 709 calculates the range of the subject on the captured image from a plurality of two-dimensional coordinates sent from the coordinate calculation unit 707, and associates the image with the unique identification information sent from the coordinate calculation unit 707. Send to output means 708. The image output unit 708 draws the output result of the subject range output unit 709 on the captured image and outputs it to the user 701.

  FIG. 8 shows an example of information stored in the sign information storage means 704. In addition to the unique identification information and the three-dimensional coordinates stored in the sign information storage means 104 of the first embodiment, a shape indicated by the three-dimensional coordinates and a flag indicating the presence or absence of concealment are added. Information relating to the unique identification information “advertisement 1” stored in the sign information storage means 704 includes three-dimensional coordinates (−60, 10, 3), (−30, 10, 30), (−30, 70, 30), (−60, 70, 0), a plane indicated by the attribute “FLAT”, and a flag for performing image concealment used in the next embodiment = “TRUE” (meaning “true”).

  As described above, according to the third embodiment, it is possible to output an image range for a specified subject on a captured image. In the third embodiment, an example in which a range indicated by a plane in a three-dimensional space is output has been described. However, by using complex three-dimensional solid information having a cube or more vertices, the shape or range of a subject can be obtained. Can be shown on the captured image.

FIG. 9 shows a captured image 900 input from the image input unit 702. The advertisement 901 in the photographed image 900 is the subject of the unique identification information “advertisement 1” stored in the sign information storage unit 704. FIG. 10 is an example of an output result of the image output unit 708 when the unique identification information “advertisement 1” is input in the subject specifying unit 703. In the figure, reference numeral 1001 denotes a two-dimensional rectangle in which a rectangle indicated by the three-dimensional coordinates stored in the sign information storage unit 704 is projected on the captured image 1000.
(Embodiment 4)
A photographing object labeling system according to Embodiment 4 of the present invention will be described.

  FIG. 11 is a configuration diagram of the photographing object labeling system according to the fourth embodiment. As shown in FIG. 11, the difference from Embodiment 3 of the photographing object labeling system of Embodiment 4 is sent from coordinate calculation means 1107 instead of subject range output means 709 of Embodiment 3. A subject range mask unit 1109 for calculating a range of a subject on a captured image from a plurality of two-dimensional coordinates and concealing the subject range by image processing, and an image output for outputting a captured image image-processed by the subject range mask unit 1109 Means 1108 is provided. In addition, the constituent elements 1101 to 1107 in the figure correspond to the constituent elements 701 to 707 in the third embodiment shown in FIG.

  The subject range mask unit 1109 calculates the range of the subject on the captured image from the plurality of two-dimensional coordinates sent from the coordinate calculation unit 1107, and conceals the obtained range of the subject on the captured image. FIG. 12 is a display example of a photographed image 1200 representing a subject range 1201 in which the region 901 indicated by the unique identification information “advertisement 1” in FIG. 8 is concealed on the input image 900 in FIG. 9 by the subject range masking means 1109.

As described above, according to the fourth embodiment, it is possible to perform concealment by performing image processing such as image cutout or mosaic on a specified subject on a captured image.
(Embodiment 5)
A photographing object labeling system according to Embodiment 5 of the present invention will be described.

  FIG. 13 is a configuration diagram of the photographing object labeling system according to the fifth embodiment. As shown in FIG. 13, the difference between the photographing object labeling system of the fifth embodiment and the fourth embodiment is that the range on the photographed image of the subject instead of the subject range mask means 1109 of the fourth embodiment. The subject image insertion means 1309 is provided to replace the image with another image. In addition, the constituent elements 1301 to 1308 in the figure correspond to the constituent elements 1101 to 1108 of the fourth embodiment shown in FIG.

  FIG. 14 shows an example of information stored in the sign information storage means 1304 of this embodiment. The difference from the example of information stored in the sign information storage means 704 used in the third to fourth embodiments shown in FIG. 8 is that a replacement image item is added.

  In FIG. 13, the subject image insertion unit 1309 calculates the range of the subject on the captured image from a plurality of two-dimensional coordinates sent from the coordinate calculation unit 1307, and uses the unique identification information sent from the coordinate calculation unit 1307. Then, a replacement image stored in the sign information storage unit 1304 in relation to the subject is taken out, and image replacement is performed in accordance with the calculated range of the subject. FIG. 15 is an example of a replacement image 1501 designated in association with “advertisement 1” in FIG. FIG. 16 is a display example of a photographed image 1600 representing a subject range 1601 in which a replacement image 1501 has been inserted by the subject image insertion unit 1309.

As described above, in the fifth embodiment, the designated subject on the captured image can be replaced with another image.
In the case of the fifth embodiment, the replacement image storage location “http: //localhost/adv.jpg” is specified in association with the unique identification information “advertisement 1” in the URL format used on the Internet. However, the replacement image data may be directly indicated.

Also, the replacement image can be inserted by simply replacing the image, performing two-dimensional rotation, enlargement, reduction, etc., accompanying three-dimensional rotation, deformation, brightness, saturation, etc. In accordance with the above, image replacement may be performed by changing the brightness, saturation, etc. of the entire replacement image or a part thereof.
(Embodiment 6)
A photographing object labeling system according to Embodiment 6 of the present invention will be described.

  FIG. 17 is a configuration diagram of the photographing object labeling system according to the sixth embodiment. The imaging object labeling system according to the sixth embodiment uses a marker that is placed in an imaging space and has direction identification means that displays unique identification information and subject direction information indicating a three-dimensional orientation from the marker to the subject. As shown in FIG. 17, there is an image input unit 1702 for a user 1701 to input a captured image and a user 1701 for indicating the position on the captured image. A subject designation unit 1703 for designating the unique identification information of the subject to be shown, a marker information storage unit 1704 for storing the unique identification information of the subject and the marker, and the related information of the marker related to the subject, and the photographed image. Marker coordinate measuring means 1705 for measuring the two-dimensional coordinates of a plurality of markers, and on the captured image of the plurality of markers captured on the captured image A marker direction measuring means 1706 for measuring the direction shown, a coordinate calculating means 1707 for calculating the coordinates of the intersection of the straight lines passing through each of the plurality of markers associated with the subject and directed in the indicated direction, and a coordinate calculating means 1707 And image output means 1708 for outputting the two-dimensional coordinates on the photographed image from the three-dimensional coordinates of the subject. Note that the user 1701, the image input unit 1702, the subject specifying unit 1703, and the image output unit 1708 are components of the user 101, the image input unit 102, the subject specifying unit 103, and the image output unit 108 of the first embodiment.

  Like the marker of the first embodiment, each marker used in the sixth embodiment has unique identification information, and the unique identification information is distinguished on the captured image by means of color, shape, barcode, light emission pattern, and the like. To do. The marker has direction indicating means for indicating a three-dimensional direction to the subject. The marker information storage means 1704 stores the unique identification information of the subject in association with the marker having the direction indicating means indicating the three-dimensional direction to the subject. FIG. 18 shows the unique identification information of the subject and the marker related to the subject stored in the sign information storage means 1704. In the example of FIG. 18, the unique identification information “marker 1” and “marker 2” of the marker are stored in association with the unique identification information “product A” of the subject.

  In FIG. 17, a marker coordinate measuring unit 1705 recognizes a marker on a captured image by image processing, measures unique identification information of each marker and two-dimensional coordinates on the captured image, and determines the captured image and the measured marker. The unique identification information and the two-dimensional coordinates are sent to the marker direction measuring means 1706. Note that the captured image sent to the marker direction measuring unit 1706 may be an image obtained by cutting out a marker portion from the input captured image or performing image processing for recognizing the marker.

  Further, the marker direction measuring unit 1706 measures the direction indicated on the captured image of the marker direction indicating unit on the captured image sent from the marker coordinate measuring unit 1705, and the unique identification information of the marker, the two-dimensional coordinates, and the direction Are sent to the coordinate calculation means 1707. The coordinate calculation means 1707 extracts a marker that matches the unique identification information of the marker related to the subject designated by the subject designation means 1703 from the marker sent from the marker direction measurement means 1706, and obtains a plurality of obtained markers. The intersection point of the straight line that passes through the coordinates on the captured image and faces the direction indicated by the marker is calculated and sent to the image output means 1708 as the coordinates of the subject.

  FIG. 19 shows an example of a captured image 1900 that the user 1701 inputs from the image input unit 1702. In the figure, reference numeral 1901 denotes a subject stored with the unique identification information of “product A” in the sign information storage unit 1704, and reference numeral 1902 denotes a subject stored with the unique identification information of “product B” in the sign information storage unit 1704. Yes, 1903 to 1906 are markers stored in the sign information storage means 1704 with the unique identification information of “markers 1 to 4”. The marker direction measuring unit 1706 measures the direction indicated by the marker direction indicating unit recognized by the marker coordinate measuring unit 1705 on the captured image 1900 and sends the captured image 1900 and the coordinates and direction of the marker to the coordinate calculating unit 1707.

  The coordinate calculation means 1707 searches for the unique identification information of the marker related to the subject from the unique identification information of the subject designated by the subject designation means 1703, and passes through the coordinates on the captured image 1900 of each of the retrieved markers. The direction instructing unit obtains the intersection of the straight lines facing the direction indicated on the captured image 1900. FIG. 20 shows the direction indication of each marker through the markers 1903 and 1904 recognized as “marker 1” and “marker 2” on the captured image 1900 when the user selects the subject 1901 as “product A”. It is an example which shows the intersection 2009 of the straight lines 2007 and 2008 which faced the direction which a means points, and the straight lines 2007 and 2008.

  As described above, according to the sixth embodiment, when the influence of aberration such as lens distortion is small on a photographed image, a projection that passes through the subject even if a straight line passing through the subject in the three-dimensional space is projected onto the photographed image. By using this property, the position of the subject on the captured image can be obtained without measuring the coordinates of the marker that guides the position of the subject and the subject by calculating the intersection of two or more straight lines passing through the subject. It is possible to construct a system that can show

  In the sixth embodiment, the sign information storage unit 1704 is mounted in the photographing target sign system, but is used in the sign information storage unit 1704 while performing wireless or wired communication or before using the system. By preparing a communication means such as downloading information or the difference between the latest information in advance, it can be placed near the subject or marker or on the Internet.

In the sixth embodiment, the marker direction instruction means uses an example in which the orientation is displayed in a shape like the marker 1904. However, by combining a plurality of markers that do not have the direction instruction means, It can be used as a single marker. FIG. 21 shows an example in which 2102 and 2104, which are a set of two markers associated with each other, show straight lines 2107 and 2108 on the photographed image 2100 and an intersection 2109 of the straight lines 2107 and 2108.
(Embodiment 7)
A photographing object labeling system according to a seventh embodiment of the present invention will be described.

  FIG. 22 is a configuration diagram of the photographing object labeling system according to the seventh embodiment. As shown in FIG. 22, the photographing object labeling system according to the seventh embodiment includes the constituent elements of the sixth embodiment, the subject movement detection means 2209 for detecting the movement of the subject, and the subject included in the marker direction instruction means. It comprises marker direction instruction information updating means 2210 for updating the direction information. Note that the constituent elements 2201 to 2208 in the figure correspond to the constituent elements 1701 to 1708 of the sixth embodiment shown in FIG.

  In FIG. 22, the subject movement detection unit 2209 detects the movement of the subject using a camera, a sensor, or the like that individually monitors the subject, and instructs the marker direction instruction information update unit 2210 with the subject's unique identification information and movement information. The marker direction instruction information update unit 2210 changes the three-dimensional direction indicated by the marker direction instruction unit stored in the sign information storage unit 2204 in association with the subject identification information sent from the subject movement detection unit 2209. I do.

As described above, according to the seventh embodiment, the same effect as that of the second embodiment can be obtained by updating the direction information indicated by the direction instruction means for each marker as the subject moves. .
(Embodiment 8)
A photographing object labeling system according to an eighth embodiment of the present invention will be described.

  FIG. 23 is a configuration diagram of the photographing object labeling system according to the eighth embodiment. As shown in FIG. 23, the photographing object labeling system according to the eighth embodiment includes guide information storage means for storing subject guide information in association with the constituent elements of the first embodiment and the unique identification information of the subject. 2310 and a guide output unit 2309 for displaying guide information of the subject in accordance with the image output unit 2308. Note that the constituent elements 2301 to 2308 in the figure correspond to the constituent elements 101 to 108 of the first embodiment shown in FIG.

  FIG. 24 shows an example of unique identification information and guide information stored in the guide information storage means 2310. In this example, the guide information of the subject unique identification information “product A-switch” is stored as “TEXT =“ Press here to turn on the power ””. “TEXT =” in the guide information indicates that the guide information is represented by a text character string.

  FIG. 25 shows an example of the operation of the guide output means 2309. When the user 2301 selects the subject identification information “product A-switch” in the subject designation unit 2303, the image output unit 2308 displays the unique identification information “stored in the sign information storage unit 2304 as shown in FIG. 25. The three-dimensional coordinates (20, −10, −30) in FIG. 2 for “product A-switch” are searched, the coordinates of the unique identification information “product A-switch” on the photographed image 200 are calculated, and “” The photographed image 2500 indicated by the text characters 2501 is displayed as “the power is turned on when this button is pressed”.

  FIG. 26 shows “” IMAGE = http: //localhost/tmp/sample.gif ”in the guide information of the unique identification information“ product B ”in FIG. This is an example of an image stored in the reference destination when link information such as “png” ”and URL is stored. FIG. 27 shows an example of the operation of the guide output unit 2309 when the user 2301 selects the subject unique identification information “product B” in the subject designation unit 2303. The image in FIG. 26 is displayed as an image 2701 on the captured image 2700.

As described above, according to the eighth embodiment, it is possible to construct a system that displays guide information such as the name and description of the subject in accordance with the position of the subject and the portion of the subject.
In the eighth embodiment, the guide information storage unit 2310 is mounted inside the imaging target sign system, but is used by the guide information storage unit 2310 while performing wireless or wired communication or before using the system. By preparing a communication means such as downloading information or the difference between the latest information in advance, it can be placed near the subject or marker or on the Internet.
(Embodiment 9)
A photographing object labeling system according to Embodiment 9 of the present invention will be described.

  FIG. 28 is a configuration diagram of the photographing object labeling system according to the ninth embodiment. As shown in FIG. 28, the photographing object labeling system of the ninth embodiment includes the constituent elements of the second embodiment and image correction means 2809 that corrects image distortion of the photographed image. In the figure, components 2801 to 2808 respectively correspond to components 1701 to 1708 of the sixth embodiment shown in FIG.

  FIG. 29 shows an example of a photographed image 2900 inputted from the image input means 2802 photographed with a wide-angle lens. In FIG. 20, the original straight line indicated by the marker direction indicating means represented by the straight line 2008 is projected onto a curved line 2901 having a distortion in FIG. The image correction unit 2809 prepares correction parameters for distortion of a captured image derived from a lens or a focal length in advance, thereby correcting the captured image so that the captured image becomes a straight line on the corrected straight line, thereby adjusting the marker coordinates. Send to measuring means 2805.

As described above, according to the ninth embodiment, a system that corrects aberration and accurately measures the position of a subject is constructed even when a captured image that is captured by a wide-angle lens or the like and has a large influence of aberration is input. be able to.
(Embodiment 10)
A photographing object labeling system according to Embodiment 10 of the present invention will be described.

  FIG. 30 is a configuration diagram of the photographing object labeling system according to the tenth embodiment. As shown in FIG. 30, the photographing object labeling system of the tenth embodiment includes the components of the first embodiment, the peripheral image input means 3009 for photographing a peripheral image around the photographing range, and the photographed image and the peripheral It comprises image composition means 3010 for creating a composite image by combining images. Note that constituent elements 3001 to 3008 in the figure correspond to the constituent elements 101 to 108 of the first embodiment shown in FIG.

  In FIG. 30, the peripheral image input unit 3009 captures images around the captured image by continuously capturing and storing images before and after capturing, zooming out the periphery of the capturing area before and after capturing, and the like. And sent to the image composition means 3010. The image composition unit 3010 combines the captured image input from the image input unit 3002 with the image around the captured image from the peripheral image input unit 3009 after correction, such as rotation, scale change, distortion correction, and the like. Is sent to the marker coordinate measuring means 3005.

As described above, according to the tenth embodiment, a system that can include markers around the captured image in the measurement even when the number of markers for guiding the position of the subject around the subject is small, such as during magnified shooting. Can be built.
(Embodiment 11)
A photographing object labeling system according to an eleventh embodiment of the present invention will be described.

  FIG. 31 is a configuration diagram of the photographing object labeling system of the eleventh embodiment. As shown in FIG. 31, the photographing object labeling system of the eleventh embodiment includes the constituent elements of the first embodiment and the marker concealing means 3109 for concealing the marker reflected on the photographed image from the photographed image by image processing. It consists of. In the figure, components 3101 to 3108 correspond to components 101 to 108 of the first embodiment shown in FIG.

  In FIG. 31, the image input unit 3102 sends the captured image input from the user 3101 to the marker coordinate measurement unit 3105 and the marker concealment unit 3109. In addition to the function of the marker coordinate measuring unit 105 of the first embodiment, the marker coordinate measuring unit 3105 sends the marker coordinates measured on the captured image to the marker concealing unit 3109. The marker concealing unit 3109 performs image processing such as blurring or changing the luminance of the peripheral portion of the marker coordinates input from the marker coordinate measuring unit 3105 in the captured image sent from the image input unit 3102. And hide. FIG. 32 shows an example of an output image 3200 showing the marker 3201 concealed by the marker concealment unit 3109 with respect to the photographed image output from the image output unit 2010.

As described above, according to the eleventh embodiment, it is possible to construct a system that can hide a marker on a captured image from the captured image.
(Embodiment 12)
A photographing object labeling system according to a twelfth embodiment of the present invention will be described.

  FIG. 33 is a configuration diagram of the photographing object labeling system of the twelfth embodiment. As shown in FIG. 33, the imaging object labeling system of the twelfth embodiment includes a marker recognition rule storage unit 3309 for storing the constituent elements of the first embodiment and marker recognition rules recognized by the marker coordinate measurement unit 3305. It consists of. In the figure, components 3101 to 3108 correspond to components 101 to 108 of the first embodiment shown in FIG.

  In FIG. 33, a marker recognition rule storage means 3309 stores a recognition rule for recognizing a marker used by the object labeling system of the present embodiment. The marker coordinate measuring unit 3305 searches for a marker recognition rule stored in the marker recognition rule storage unit 3309 in accordance with the type of marker used at the shooting location, and uses the corresponding marker recognition rule to perform shooting. Recognize the marker on the image.

  As described above, according to the twelfth embodiment, by switching the marker recognition rules, a system is constructed that can select markers according to shooting conditions such as indoor and outdoor lighting conditions and the arrangement of shooting objects. be able to.

  When displaying information related to a subject on a photographed image of a portable terminal having a photographing means, the photographing object labeling system of the present invention can easily and accurately position the subject on the photographed image in a short time. A TV equipped with a portable terminal having photographing means such as a digital camera, a cellular phone with a digital camera, an electronic notebook with a digital camera, and an electronic album for reproducing digitally photographed images. Is applicable.

Configuration diagram of imaging object labeling system according to Embodiment 1 of the present invention The figure which shows the picked-up image in the imaging | photography target object labeling system of Embodiment 1. FIG. The figure explaining operation | movement of the to-be-designated means in the imaging | photography target object labeling system of Embodiment 1 The figure which shows the example of the information stored in the label | marker information storage means in the imaging | photography object label | marker system of Embodiment 1 The figure explaining operation | movement of the image output means in the imaging | photography target object labeling system of Embodiment 1 Configuration diagram of imaging object labeling system of Embodiment 2 of the present invention Configuration diagram of imaging object labeling system of Embodiment 3 of the present invention The figure which shows the example of the information stored in the label | marker information storage means in the imaging | photography object label | marker system of Embodiment 3 The figure which shows the picked-up image in the imaging | photography target object labeling system of Embodiment 3. The figure explaining operation | movement of the to-be-photographed object range output means in the imaging | photography target object labeling system of Embodiment 3. FIG. Configuration diagram of imaging object labeling system of Embodiment 4 of the present invention The figure explaining operation | movement of the to-be-photographed object range mask means in the imaging | photography target object labeling system of the same Embodiment 4. FIG. Configuration diagram of imaging object labeling system of Embodiment 5 of the present invention Example of information stored in sign information storage means in the photographing object sign system of Embodiment 5 Example of replacement image stored in sign information storage means in photographing object sign system of embodiment 5 The figure explaining operation | movement of the to-be-photographed image insertion means in the photographic subject labeling system of Embodiment 5. Configuration diagram of photographing object labeling system of Embodiment 6 of the present invention Example of information stored in sign information storage means in photographing object sign system of embodiment 6 The figure which shows the picked-up image in the imaging | photography target object labeling system of Embodiment 6. The figure which shows operation | movement of the coordinate calculating means in the imaging | photography target object label | marker system of Embodiment 6. The figure which shows different mounting of the direction instruction | indication means of the marker in the imaging | photography target object labeling system of Embodiment 6 Configuration diagram of photographing object labeling system of Embodiment 7 of the present invention Configuration diagram of imaging object labeling system of Embodiment 8 of the present invention Example of information stored in sign information storage means in photographing object sign system of embodiment 8 The figure explaining operation | movement of the guide output means in the imaging | photography target object labeling system of the same Embodiment 8. FIG. Example of image stored in sign information storage means in photographing object sign system of embodiment 8 The figure explaining operation | movement of the guide output means in the imaging | photography target object labeling system of the same Embodiment 8. FIG. Configuration diagram of photographing object labeling system of Embodiment 9 of the present invention The figure explaining the picked-up image with a distortion in the imaging | photography target object labeling system of Embodiment 9. Configuration diagram of imaging object labeling system of Embodiment 10 of the present invention Configuration diagram of photographing object labeling system of Embodiment 11 of the present invention The figure explaining operation | movement of the marker concealment means in the imaging | photography target object labeling system of Embodiment 11 Configuration diagram of imaging object labeling system of Embodiment 12 of the present invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 User 102 Image input means 103 Subject designation means 104 Marking information storage means 105 Marker coordinate measurement means 106 Coordinate conversion parameter calculation means 107 Coordinate calculation means 108 Image output means 200 Captured image 201 Marker 202 Product A
609 Coordinate change detection means 610 Three-dimensional coordinate update means 709 Subject range output means 1109 Subject range mask means 1309 Subject image insertion means 1706 Marker direction measurement means 2209 Subject movement detection means 2210 Marker direction instruction information update means 2309 Guide output means 2310 Guide information Storage means 2809 Image correction means 3009 Peripheral image input means 3010 Image composition means 3109 Marker concealment means 3309 Marker recognition rule storage means

Claims (12)

Using a plurality of markers associated with a subject, a photographing object labeling system that calculates and outputs coordinates on a photographed image of the subject,
Image input means for inputting the captured image;
Subject specifying means for specifying the subject using the unique identification information of the subject;
Label information storage means for storing a subject to be designated by the subject designation means and three-dimensional coordinates of the plurality of markers;
Marker coordinate measuring means for measuring two-dimensional coordinates of a plurality of markers reflected on the captured image;
Coordinates for calculating two-dimensional coordinates in which arbitrary three-dimensional coordinates are projected on the photographed image using the three-dimensional coordinates of the plurality of markers reflected on the photographed image and the two-dimensional coordinates on the photographed image. Coordinate conversion parameter calculation means for calculating the parameters of the conversion formula;
Coordinate calculation means for calculating the two-dimensional coordinates of the subject projected on the photographed image from the three-dimensional coordinates of the subject using the coordinate conversion formula;
An imaging object labeling system comprising: an image output means for drawing a position of a subject on the captured image on the captured image. The imaging object labeling system according to claim 1,
Coordinate change detecting means for detecting a change in the three-dimensional coordinates of the subject;
An imaging object labeling system comprising: a three-dimensional coordinate updating unit that updates the three-dimensional coordinates of the subject stored in the marker information storage unit according to a detection result of the coordinate change detection unit. The imaging object labeling system according to claim 1 or 2,
A photographing object labeling system comprising subject range output means for representing a subject on a photographed image by representing the subject with a combination of a plurality of three-dimensional coordinates. The imaging object labeling system according to claim 1 or 2,
A subject to be photographed, comprising subject range masking means for concealing an output range on the photographed image of the subject shown by representing the subject with a combination of a plurality of three-dimensional coordinates from the photographed image by image processing Object labeling system. The imaging object labeling system according to claim 1 or 2,
A photographing object labeling system comprising: subject image insertion means for replacing an output range of a subject on a photographed image indicated by representing the subject with a combination of a plurality of three-dimensional coordinates with another image. A photographing object labeling system that calculates and outputs coordinates on a photographed image of the subject using a plurality of markers associated with the subject and having direction indicating means indicating a three-dimensional direction to the subject,
Image input means for inputting the captured image;
Subject specifying means for specifying the subject using the unique identification information of the subject;
Label information storage means for storing a subject to be designated by the subject designation means and three-dimensional coordinates of the plurality of markers;
Marker coordinate measuring means for measuring two-dimensional coordinates of a plurality of markers reflected on the captured image;
Marker direction measuring means for measuring the direction indicated by the plurality of marker direction indicating means shown on the captured image, on the captured image;
A coordinate calculation that calculates, as the two-dimensional coordinates of the subject, the two-dimensional coordinates of the intersection of straight lines that pass through each of the plurality of markers associated with the subject on the captured image and face the direction indicated by the direction indicating means of each marker Means,
An imaging object labeling system comprising: an image output means for drawing a position of a subject on the captured image on the captured image. The imaging object labeling system according to claim 6,
Subject movement detection means for detecting movement of the subject;
An imaging object labeling system comprising: marker direction instruction information updating means for updating the direction instruction means for the marker based on a detection result of the subject movement detection means. The imaging object labeling system according to claim 1,
Guide information storage means for storing guide information of the subject;
A photographing object labeling system comprising: guide output means for displaying guide information of a subject in the guide information storage means in association with two-dimensional coordinates of the subject on the photographed image. A photographing object labeling system according to claim 1,
An imaging object labeling system comprising: an image correction unit that outputs a corrected image obtained by correcting lens distortion of the captured image input from the image input unit. The imaging object labeling system according to claim 1,
A peripheral image input means for capturing and inputting a peripheral image around the shooting range;
Image synthesizing means for creating a composite image obtained by combining the photographed image input from the image input means and the peripheral image from the peripheral image input means;
A photographing object labeling system comprising: composite image input means for inputting a composite image from the image composition means. A photographing object labeling system according to claim 1,
An imaging object labeling system comprising marker hiding means for hiding the marker reflected on the photographed image input from the image input means from the photographed image by image processing. The imaging object labeling system according to claim 1,
An imaging object marking system comprising marker recognition rule storage means for storing a marker recognition rule recognized by the marker coordinate measuring means.

Images