JP2009140402A - Information display device, information display method, information display program, and recording medium with recorded information display program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To display additional information while superposing it on a photographed image. <P>SOLUTION: Photographic information on a photographic position, a photographic direction, etc., of photography and map information including the photographic position are acquired. Consequently, it is possible to calculate what building is photographed in which part of the photographed image so that additional information related to the photographed building is acquired, and can be displayed while being superimposed on the photographed image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for adding information to a captured image.

  There has been proposed a system for adding information to an image using an identifier that can be recognized by a photographing apparatus such as a QR code. The identifier is drawn in a pattern that is easy to find when placed in real space, and by placing this identifier in real space, it is possible to virtually add information to the position of the identifier in the image taken by the camera Become. That is, by placing an identifier around a certain object, it is possible to add information unique to the identifier to the video of that object.

A method using blinking of visible light as an identifier has been proposed (see Patent Document 1 and Non-Patent Document 1). In this method, a device that emits visible light is installed on an object to which information is to be added. The imaging device captures an object and also receives information transmitted by the blinking interval of visible light. As a result, the received information can be added to the photographed object. Since visible light reaches far, it is possible to acquire information added to a distant object. For example, by installing this light emitting device in a building, it is possible to add information to an image of the building.
JP 2005-56054 A Nobuyuki Matsushita, 4 others, “ID Cam: Image sensor that can acquire scene and ID simultaneously”, Interaction 2002, 2002, p.9-16

  Thus, in order to virtually add information to an object, it is necessary to install a device and an identifier on the object. Further, in order to obtain information added to an object, it is necessary to take a picture including its identifier.

  However, it is impractical to place identifiers or the like on all objects to which information is to be added, or to place identifiers in all directions of an object so that information can be acquired from all directions.

  The present invention has been made in view of the above, and an object thereof is to display additional information superimposed on a captured image without arranging an identifier on the object.

  An information display device according to a first aspect of the present invention acquires imaging information for capturing an image, position measuring means for measuring a capturing position and a capturing direction when an image is captured, and map information including object feature points. Map information acquisition means; first feature calculation means for calculating a vertical feature line in an image based on an angle formed by lines connecting a shooting position and a feature point of an object in the shooting direction on the map information; And position specifying means for specifying the position of the object shown in the image based on the feature line.

  In the present invention, the map position including the shooting position and shooting direction when the image is shot and the feature point of the object is acquired, and the angle formed by the lines connecting the feature point of the object from the shooting position on the map information By using this angle and calculating the feature line in the vertical direction that divides the captured image in the horizontal direction, it is possible to specify the object that appears between the feature lines.

  In the information display device, the position measuring means further measures an inclination in the vertical direction with respect to the photographing direction, and includes an altitude measuring means for measuring an altitude when the image is taken, and a feature line for each feature line. And a second feature calculation means for calculating a feature point on the feature line based on the distance, inclination and altitude to the object corresponding to.

  In the present invention, the boundary between the object and the ground can be specified by measuring the altitude and the inclination.

  In the above information display device, additional information acquisition means for acquiring additional information corresponding to the object whose position in the image is specified from a database storing additional information about the object, and displaying the image, the position of the object in the image Display means for displaying additional information in correspondence with each other.

  In the present invention, by acquiring the additional information corresponding to the object whose position in the photographed image is specified, it is possible to superimpose and display the additional information on the photographed image. The additional information is, for example, an advertisement related to an object (such as a building) or a road guide.

  In the above information display device, rotation angle detecting means for rotating the image or the feature line to overlap each other and detecting the rotation angle when there are many portions where the feature points of the object shown in the image overlap the feature line It is characterized by having.

  In the present invention, the photographed image and the calculated feature line are rotated and overlapped little by little, and the rotation angle when there are many portions where the feature points of the object and the feature line appear in the image overlap is detected. Thus, it is possible to calculate the rotation angle of the information display device when an image is taken.

  According to a second aspect of the present invention, there is provided an information display method comprising: a step of photographing an image by a photographing unit; a step of measuring a photographing position and a photographing direction when the image is photographed by a position measuring unit; and a map information obtaining unit. An image based on an angle formed by lines connecting the shooting position and the feature point of the object in the shooting direction on the map information by the step of obtaining the map information including the feature point of the object and the first feature calculation means. And a step of specifying a position of an object shown in an image based on the feature line by a position specifying unit.

  An information display program according to a third aspect of the present invention causes a computer to function as each means included in the information display device.

  A recording medium according to a fourth aspect of the present invention records the information display program.

  According to the present invention, it is possible to superimpose and display additional information on a captured image without placing an identifier on the object.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing the configuration of the information display device in the present embodiment. The information display device 1 shown in FIG. 1 includes an imaging unit 11, a position measuring unit 12, an orientation measuring unit 13, an inclination measuring unit 14, an altitude measuring unit 15, a communication unit 16, a vertical feature point specifying unit 17, and a horizontal feature point specifying unit. 18 and a display unit 19. The information display device 1 may be configured by a computer including an arithmetic processing device, a storage device, a memory, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the information display device 1, and can be recorded on a recording medium such as a CD-ROM, a magnetic disk, an optical disk, or a semiconductor memory, or can be provided through a network. Details of each part will be described below.

  The photographing unit 11 photographs a landscape video and inputs an image. The image input by the imaging unit 11 is not limited to a still image and may be a moving image.

  The position measurement unit 12, the orientation measurement unit 13, the tilt measurement unit 14, and the altitude measurement unit 15 measure the shooting position, the shooting direction, the tilt of the information display device 1, and the shooting altitude when the shooting unit 11 takes an image. The photographing position is measured using GPS (Global Positioning System), and the photographing direction and inclination are measured using a biaxial sensor or a triaxial sensor. Altitude is measured using an altimeter.

  The communication unit 16 is connected to the map information database 20 and the additional information database 30. The communication unit 16 acquires map information from the map information database 20 and acquires additional information related to the captured image from the additional information database 30.

  The vertical feature point specifying unit 17 specifies a vertical feature line that appears in an image shot based on the angle of view, the shooting position, the shooting direction, and the map information of the shooting unit 11. The vertical feature line is a characteristic line that appears in the vertical direction (vertical direction) in a captured image such as a corner of a building, for example.

  The horizontal feature point specifying unit 18 specifies a horizontal feature point that appears in the captured image based on the angle of view, the tilt, the shooting altitude, and the vertical feature line of the shooting unit 11. The feature point in the horizontal direction is a feature point that appears in the horizontal direction (left-right direction) in a photographed image such as a boundary line between a building and the ground. Details of the processing of the vertical feature point specifying unit 17 and the horizontal feature point specifying unit 18 will be described later.

  The display unit 19 superimposes the additional information acquired from the additional information database 30 on the image captured by the imaging unit 11 and displays it. The additional information specifies the object (building, landmark, etc.) that appears in the image captured based on the feature lines and feature points detected by the vertical feature point specifying unit 17 and the horizontal feature point specifying unit 18, and the specified object. Overlapping display corresponding to.

  FIG. 2 is a schematic diagram showing how the information display device 1 is used. As shown in the figure, when a user photographs a landscape including a building 201 with the information display device 1, the information display device 1 identifies the building 201 in the image and acquires additional information related to the building. indicate.

  Next, a processing flow of the information processing apparatus will be described. FIG. 3 is a flowchart showing a processing flow of the information display apparatus 1.

  First, a landscape is photographed by the photographing unit 11 to obtain an image (S301).

  Subsequently, the position measurement unit 12, the direction measurement unit 13, the tilt measurement unit 14, and the altitude measurement unit 15 measure the shooting position, shooting direction, tilt, and altitude at the time of shooting (S302).

Subsequently, the vertical feature point specifying unit 17 calculates a vertical feature line (S303). First, the vertical feature point specifying unit 17 acquires a shooting position and a shooting direction from the position measurement unit 12 and the orientation measurement unit 13. Then, map information including the shooting position is acquired from the map information database 20 via the communication unit 16. As shown in FIG. 4, the map information includes a shape such as a contour of a building when viewed from above. A to K on FIG. 4 indicate buildings, and a point O indicates a shooting position. By taking into consideration the shooting position, shooting direction, and angle of view ψ _x , it is possible to estimate the shot area. The angle of view ψ _x is acquired from the imaging unit 11.

And the vertical feature point specific | specification part 17 calculates the positional relationship of the building in the image image | photographed using map information. As shown in FIG. 5, the positional relationship between the buildings is as follows. A line is drawn from the point O on the map information to the corner of the building in the region of the field angle ψ _x around the shooting direction. It calculates using angle _ai which the reference line to show makes. If the resolution of the photographed image is a x × y pixels, the width x _i of the i-th feature line in the photographed image can be expressed by the following equation.

Like the image shown in FIG. 6, the position of a feature point such as a corner of a building can be specified by drawing a vertical line (vertical feature line) for each width x _i calculated by the above formula. In FIG. 6, a solid line corresponds to a boundary between a building and another building, and a dotted line corresponds to a corner of the building. This makes it possible to estimate which building is captured between the solid lines in FIG. 6 in the captured image.

  Subsequently, the lateral feature point specifying unit 18 calculates a lateral feature point (S304). The lateral feature point specifying unit 18 acquires the inclination θ and the altitude h from the inclination measuring unit 14 and the altitude measuring unit 15. As shown in FIG. 7, the inclination θ indicates how much the information display device 1 is inclined in the vertical direction. That is, the inclination θ indicates how much the depth direction of the captured image is inclined with respect to the horizontal direction, and the altitude h indicates how far the information display device 1 is from the ground surface.

  The feature point in the horizontal direction is obtained for each feature line in the vertical direction calculated by the vertical feature point specifying unit 17. That is, feature points in the horizontal direction are calculated for each solid line and dotted line in FIG.

As shown in FIG. 8, when the altitude h is sufficiently high, such as when the user takes a picture from the roof of the building, only the building 201 is shown, and no lateral feature points such as the ground appear. Specifically, when the following equation holds, no feature point in the horizontal direction appears in the feature line in the vertical direction.

Here, k indicates the distance from the shooting position to the building 201, and ψ _y indicates the angle of view. The distance k can be calculated by using map information.

On the other hand, as shown in FIG. 9, when the user photographs from the ground and the altitude h is low, the ground may appear in the photographed image, and the boundary between the building and the ground is detected as a lateral feature point. Can do. Specifically, it is when the following equation holds.

At this time, the feature point in the horizontal direction can be calculated for each feature line in the vertical direction using the following equation.

Here, y _i1 is a portion corresponding to a building on the image, and y _i2 is a portion corresponding to a road on the image. As shown in FIG. 10, the vertical feature line is divided into y _i1 and y _i2, and this division point (characteristic point in the horizontal direction) is connected for each building, so that as shown in FIG. The boundary with the ground can be calculated.

  Subsequently, it is detected how much the captured image is rotated with respect to the axis in the depth direction of the image (S305). The rotation angle can be detected using a triaxial sensor. A method for detecting the rotation angle without using the triaxial sensor will be described later.

  Subsequently, the display unit 19 acquires additional information related to the detected building from the additional information database 30 via the communication unit 16 (S306), and matches the position of the building calculated above as shown in FIG. The additional information is displayed over the captured image (S307). When the captured image is rotated, the display position of the additional information is rotated according to the rotation angle.

  As described above, since the position of the building in the captured image can be estimated, it is possible to virtually display the advertisement on the captured image by registering the advertisement of the building tenant in the additional information database 30. It becomes. Also, by registering landmark information and road information in the additional information database 30, it can be used for road guidance and the like.

  Next, a method for obtaining the rotation angle of the screen without using a triaxial sensor will be described. First, an edge image is obtained by emphasizing a portion of a photographed image where a change in color or brightness that is a characteristic point of a building is high. As shown in FIG. 13, the rotation angle between the edge image 501 and the image in which the vertical feature line is calculated is superimposed while gradually shifting the angle from 0 degrees to 360 degrees, and one pixel where the feature line and the edge overlap is obtained. The most frequent angle is adopted as the rotation angle. Note that when an image for which a feature line is calculated is rotated, new buildings that are not estimated may be reflected in the corners of the captured image, but these buildings are not considered.

  Further, when the map information includes building height information, a building 601 existing behind the building H can be detected as shown in FIG. The rooftop portion of the building can be calculated in the same manner as the method using the lateral feature point specifying unit 18 described above.

  Hereinafter, a method of using the information display device 1 will be briefly described.

  Street advertising is limited in location, and few companies can advertise street advertising. Therefore, when a certain building is photographed by the information display device 1, an advertisement can be virtually displayed on the building.

  In addition, when using a map for route guidance, the map and the actual landscape must be associated in the head. Therefore, by photographing the scenery in front of the information display device 1, it is possible to display an arrow superimposed on the road portion of the photographed image or to display a mark on a target building. Thereby, since the guide information is displayed only by the user taking a picture with the information display device 1, it is possible to provide route guidance that is easy for the user to understand.

  Furthermore, by inputting information such as the height of the building and the height of the mountain into the map information, the information display device 1 can be installed on an observation deck and the like, and information can be added to a conspicuous building or mountain. It becomes possible.

  The present invention can also be applied to a communication tool for adding information to the additional information database 30. For example, by storing impressions and word-of-mouth for tenants of a building in the additional information database 30, it is possible to view the impressions of other people by photographing the building with the information display device 1.

  As described above, according to the present embodiment, by acquiring shooting information such as the shooting position and shooting direction at the time of shooting and map information including the shooting position, which building around which shot image is taken. Therefore, it is possible to calculate whether or not the captured image includes the additional information regarding the captured building from the additional information database 30 and to display the additional information on the captured image.

  According to the present embodiment, the boundary between the photographed building and the ground can be calculated by acquiring the vertical inclination and altitude at the time of photographing, so the road guidance is superimposed on the road portion of the photographed image. Can be displayed.

  According to the present embodiment, the edge image 501 is obtained and superimposed with the image obtained by calculating the vertical feature line gradually shifting the rotation angle, and the angle with the largest number of pixels where the feature line and the edge overlap is obtained. By adopting the rotation angle, the rotation angle of the screen can be obtained without using a triaxial sensor.

It is a block diagram which shows the structure of the information display apparatus in one Embodiment. It is a schematic diagram which shows a mode that the said information display apparatus is utilized. It is a flowchart which shows the flow of operation | movement of the said information display apparatus. It is a figure which shows the map information used with the said information display apparatus. It is a figure which shows a mode that the line was drawn to the feature point of the object on map information. It is a figure which shows the characteristic line of the vertical direction computed from the said map information. It is explanatory drawing for demonstrating inclination and an altitude. It is a figure which shows a mode that the feature point of a horizontal direction is calculated with the distance, inclination, and height to an object. It is another figure which shows a mode that the feature point of a horizontal direction is calculated with the distance to an object, inclination, and height. It is a figure which shows the feature point of the calculated horizontal direction. It is a figure which shows the position of the object which the said information display apparatus calculated. It is a figure which shows a mode that additional information was superimposed and displayed on the image | photographed image. It is a figure which shows a mode that the image | photographed image is rotated and a rotation angle is calculated | required. It is a figure which shows a mode that the position of the object was calculated using the height information of an object.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Information display apparatus 11 ... Imaging | photography part 12 ... Position measurement part 13 ... Orientation measurement part 14 ... Inclination measurement part 15 ... Altitude measurement part 16 ... Communication part 17 ... Vertical feature point specific | specification part 18 ... Horizontal feature point specific | specification part 19 ... Display Part 20 ... Map information database 30 ... Additional information database

Claims (10)

Photographing means for photographing an image;
Position measuring means for measuring a shooting position and a shooting direction when the image is shot;
Map information acquisition means for acquiring map information including feature points of the object;
First feature calculating means for calculating a vertical feature line in the image based on an angle formed by lines connecting the shooting position and the feature point of the object in the shooting direction on the map information;
Position specifying means for specifying the position of the object in the image based on the feature line;
An information display device comprising: The position measuring means further measures the vertical tilt with respect to the shooting direction,
Altitude measuring means for measuring altitude when the image is taken;
Second feature calculation means for calculating, for each feature line, a feature point on the feature line based on a distance to the object corresponding to the feature line, the inclination, and the altitude;
The information display device according to claim 1, further comprising: Additional information acquisition means for acquiring the additional information corresponding to the object for which the position in the image has been identified from a database storing additional information related to the object;
Display means for displaying the image and displaying the additional information in correspondence with the position of the object in the image;
The information display device according to claim 1, further comprising:   Rotating angle detecting means for rotating the image or the feature line to overlap each other and detecting a rotation angle when there are many portions where the feature points of the object shown in the image overlap the feature line The information display device according to any one of claims 1 to 3. A step of photographing an image by photographing means;
Measuring a photographing position and a photographing direction when the image is photographed by a position measuring means;
Obtaining map information including feature points of the object by map information obtaining means;
A step of calculating a vertical feature line in the image based on an angle formed by lines connecting the shooting position and the feature point of the object in the shooting direction on the map information by the first feature calculation means. When,
A step of specifying a position of the object in the image based on the feature line by position specifying means;
An information display method characterized by comprising: The position measuring means further measures the vertical tilt with respect to the shooting direction,
A step of measuring an altitude when the image is taken by an altitude measuring means;
Calculating a feature point on the feature line based on the distance to the object corresponding to the feature line, the inclination, and the altitude by the second feature calculation unit;
The information display method according to claim 5, further comprising: Acquiring the additional information corresponding to the object whose position in the image is specified from a database in which additional information about the object is stored by an additional information acquisition unit;
Displaying the image by display means, and displaying the additional information in correspondence with the position of the object in the image;
The information display method according to claim 5 or 6, characterized by comprising:   The step of rotating either the image or the feature line by the rotation angle detection means and superimposing each of them, and detecting the rotation angle when there are many portions where the feature points of the object shown in the image overlap the feature line The information display method according to claim 5, further comprising:   The information display program for functioning a computer as each means which the information display apparatus as described in any one of Claims 1 thru | or 4 has.   5. A recording medium on which an information display program for causing a computer to function as each unit included in the information display device according to claim 1 is recorded.

Images