JP5001400B2 - Shield removal display device - Google Patents

Description

  The present invention relates to a display device that estimates an image in a state where a shielding object is removed, and combines and displays the image with a background image.

  In the conventional technology to specify an object in a captured image as a shielding object, and to estimate and generate an image captured without the shielding object, a range of distance to the shielding object is specified, and an object in the range is selected. Designated as a shield (see, for example, Patent Document 1). In some cases, the user determines the position of an object in a photographed image and designates a shielding object to be removed. (For example, refer to Patent Document 2). In addition, there are some that specify a distance to an object to be displayed and display only an image of the object at the distance. (For example, refer nonpatent literature 1).

JP 2008-117305 A JP-A-11-120352

Flaneur: digital see-through telescope (Keio University Hiroshi Kato et al.), International Conference on Computer Graphics and Interactive Techniques, ACM SIGGRAPH ASIA 2008, SESSION: Emerging technologies, Pages 43-43 Analysis and Machine Intelligence, 22 (11): 1330-1334, 2000., "A flexible new technique for camera calibration". IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (11): 1330-1334, 2000. Basic mathematics for CAD / CG, Shizuo Shimada, Mathematics series 7 in the Internet age, Kyoritsu Shuppan, 2000.

However, the conventional configuration is effective when the objects to be observed and the objects to be removed are simple shapes and the number is small, but it is directly visible when walking on a complex street with many buildings. If you ’re looking for a destination you ca n’t see easily, or if you ’re looking to find a type of store on the street where you ca n’t see the surrounding scenery directly, such as a restaurant,
Simple specification of distance alone does not remove the obstructions as intended by the user, so images as intended are often not obtained, and the operation becomes complicated when all the objects to be removed are designated. There was a problem of causing problems in operability.

  In addition, as a technique for displaying those images on a screen directed in the target direction when it is impossible to capture a background image or an object image from the shooting location, a technique such as Non-Patent Document 1 is disclosed. However, in this case as well, only a technique for acquiring and displaying an image of a position designated by a direction and a distance from a server is disclosed, and when used for continuous operation on a street having a complicated shape, the operation is disclosed. And visibility are insufficient. Further, in Non-Patent Document 1, only a landscape that cannot be directly visually recognized is displayed, and it is difficult to grasp the relationship with the landscape that can be directly visually recognized by the user, and it is difficult for the user to recognize the distance and direction to the target object. Had.

  The present invention solves the above-described conventional problems, and can be used to specify the removal of the shielding object intended by the user with a simple operation, and to easily recognize the distance and direction to the object. An object of the present invention is to provide an object removal display device.

  In order to solve the above-described conventional problems, the obstruction removal display device of the present invention has a device position, a direction detection unit, and a surrounding map information acquisition unit. When a candidate for a road section that can display a background image is detected by removing the shielding object, the background image of the section selected by the user is acquired and displayed in a captured image, and the user changes the direction of the device Next, a section that is continuous with the section in which the background image was displayed immediately before is selected based on the map information, set as the next section to display the background image, and should be removed to display the background image in that section. The object is determined, and the removal of the shielding object is displayed again. With this configuration, a background image of a continuous section is displayed as the direction of the apparatus is changed.

  According to the obstruction removal display device of the present invention, when a landscape image is displayed while removing obstructions in an area where there are roads or buildings with complicated shapes, the user only needs to change the direction of the device. It is possible to continuously select a road section that meets the intention of the vehicle, and to easily recognize the distance and direction.

Configuration diagram of a shield removal display device according to an embodiment of the present invention The figure which shows arrangement | positioning of the building and viewpoint in embodiment of this invention The figure which shows the example of a screen in embodiment of this invention The figure which shows the example of a screen in embodiment of this invention The figure which shows the example of a screen in embodiment of this invention The figure which shows the example of a screen in embodiment of this invention The figure which shows arrangement | positioning of the building in the embodiment of this invention, a viewpoint, and a background image

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram of a shield removal display device according to an embodiment of the present invention. In FIG. 1, a position detection unit 101 detects the position of the apparatus. The map information acquisition unit 102 acquires map information around the position obtained from the position detection unit 101. The direction detection unit 103, the elevation angle detection unit 104, and the tilt angle detection unit 105 detect the direction, the elevation angle, and the horizontal tilt of the imaging unit 107, respectively. The focal length detection unit 106 detects the focal length of the lens of the photographing unit 107. The imaging unit 107 captures an image. The section selection unit 108 is configured so that when the operation of the apparatus is started, the photographing unit 107 determines the position detected by the position detection unit 101, the map information detected by the map information acquisition unit 102, and the direction detected by the direction detection unit 103. A section of the road in the direction of shooting is selected from the map and set in the section setting unit 112. This selection method will be described later. The display section switching determination unit 109 is set in the section setting unit 112 based on the position detected by the position detection unit 101, the map information detected by the map information acquisition unit 102, and the direction detected by the direction detection unit 103. It is determined whether or not the direction of the photographing unit 107 points within the section. When the display interval switching determination unit 109 determines that the center of the shooting direction is outside the range of the interval set in the interval setting unit 112, the continuous interval determination unit 110 determines the next interval for displaying the background image. To do. This determination method will be described later. The section change unit 111 accepts an operation when the user instructs display of a background image of a section of another road based on the background screen displayed on the display unit 116, and the section set in the section setting unit And the changed road section is obtained and set in the section setting unit 112. Details of the method of changing this section will be described later. The removal target determination unit 113 takes out the road section set in the section setting unit 112, and displays the position and map information detected by the position detection unit 101 for the shielding object to be removed from the captured image when displaying the background image of the road. The determination is made using the map information detected by the acquisition unit 102 and the direction detected by the direction detection unit 103. This determination method will be described later. The background image acquisition unit 114 acquires an image of the background of the shielding object determined to be removed by the removal target determination unit 113. This acquisition method will be described later. The fitting composition unit 115 uses the direction detected by the direction detection unit 103, the elevation angle detected by the elevation angle detection unit 104, the tilt angle detected by the tilt angle 105 detection unit, and the focal length detected by the focal length detection unit 106. The background image acquired by the image acquisition unit 114 is combined with the image captured by the image capturing unit 107 by matching the position and size of the image and the image is generated. This fitting and synthesizing method will be described later. The display unit 116 displays the image generated by the fitting composition unit 115.

  FIG. 2 is a diagram illustrating an arrangement of buildings and viewpoints assumed when an example of a screen to be described later is described. In FIG. 2, buildings A 201 A to E 201 E, buildings A ′ 202 A to E E 202 E are buildings related to screen display, sections A to H, sections U to Z are road sections related to screen display, and photographing unit 107. This is a section of a road that is relevant when a method for selecting a section for displaying a background image next is described. Directions A to G indicate directions to be used when explaining the method for determining the switching of the road section displaying the background image. The end points A and B indicate the end points of the section C. The viewpoint 203 is a position when photographing the building group.

  FIG. 3 is an example of a screen in the embodiment of the present invention. This screen example is a screen example when the building A 201A to the building E 201E are photographed from the viewpoint 203 of FIG.

  4A to 4C are screen examples in the embodiment of the present invention. By performing the processing described later using the above-described configuration, the direction of the photographing unit 107 is changed from the direction A to the display unit 116 as the direction of the photographing unit 107 is moved from the direction A to the direction E in FIG. When the screen in FIG. 4A is between the direction B and when the screen in FIG. 4B is between the direction C and the direction D when the screen in FIG. 4C is displayed.

  Next, the operation of the section selection unit 108 will be described. A description will be given assuming that the photographing unit 107 faces the direction F at the start of the operation of the apparatus. In this case, the section selection unit 108 obtains a straight line F extending in the direction F from the viewpoint 203 in FIG. 2 based on the position obtained from the position detection unit 101 and the direction obtained from the direction detection unit 103. Furthermore, the section selection unit 108 extracts a section of a road on the surrounding map from the map information obtained from the map information acquisition unit 102. At this time, in the map information, information on road sections divided in block units is stored as coordinates at both ends, and the section selection unit 108 extracts road sections as line segments based on the coordinates at both ends. Shall be able to. Further, for the sake of simplicity, the description here assumes that the block and the building have a one-to-one correspondence. When performing control in units of buildings rather than blocks, it is possible to subdivide the road unit into sections facing the building.

  The section selection unit 108 extracts the section of the road on the map that intersects with the straight line F. In the example of FIG. 2, section B, section E, and section F are extracted. The section selection unit 108 obtains intersections with the respective sections and distances between the respective intersections and the viewpoint 203. Next, a section with the second shortest distance between the viewpoint 203 and the intersection is selected. This selection criterion may be settable. In practice, it is considered that the usage form in which the building at the forefront is removed as a shielding object is common, so in this embodiment, the background image is used as the second closest section in order to remove the foreground shielding object. It demonstrates as what is selected as an area for displaying.

  Next, the operation of the removal target determination unit 113 will be described. The removal target determination unit 113 extracts the section set by the section selection unit 108, the continuous section determination unit 110, or the section change unit 111 from the section setting unit 112. Next, the removal target determination unit 113 extracts map information from the map information acquisition unit 102. Next, the removal target determination unit 113 extracts a building including a part or all of the triangle with the position as one vertex and the extracted section as one side from the map information, and removes those buildings. Judge as a shield to be. In FIG. 2, when the section B is set as the section, the building C 201 </ b> C including a part in the triangle connecting the section B and the viewpoint 203 is determined as the shielding object to be removed. When the section C is set as the section, the building C201C and the building D201D that partially include the triangle connecting the section C and the viewpoint 203 are determined as the shielding objects to be removed.

  Next, the operation of the background image acquisition unit 114 will be described with reference to FIG. 2 and FIG. Here, it is assumed that the section C is set in the section setting unit 112 and the building C201C and the building D201D are designated by the removal target determination unit 113 as shielding objects to be removed. In addition, the road section around each building is obtained from the map information.

  First, when a plurality of shielding objects are designated, the background image acquisition unit 114 obtains a section around the shielding objects, and a section sandwiched between the shielding objects, that is, a plurality of shielding objects is common. Exclude surrounding sections you have. In the case of the building C201C and the building D201D, since the section V is shared as a surrounding section, the section V is excluded.

  Next, the background image acquisition unit 114, among all the remaining sections, that is, all the sections around the shielding object to be removed, the section on the near side of the shielding object, that is, the foreground when viewed from the viewpoint 203. Exclude sections. Specifically, one point in each section is selected, a straight line connecting the viewpoint 203 and the selected point is obtained, and the intersections with all the other sections having the intersection with the straight line are within the previously selected section. If it is farther than the point, that is, if the point in the previously selected section is in the foreground, that section is excluded as the forefront section. In FIG. 2, when the section C is set and the building C201C and the building D201D are selected as the shields to be removed, the section E and the section G are excluded as the forefront section, and as a result, the section B and the section C, section T, and section X are designated as the sections from which the background image should be acquired.

  Next, the background image acquisition unit 114 acquires background images corresponding to the sections B, C, T, and X. In order to acquire the background image, the background image acquisition unit 114 sets a shooting point at the center of the section, and acquires a landscape shot image having a shooting direction in a direction orthogonal to the section from the server. In the current general landscape image providing service, a landscape image can be acquired by specifying latitude, longitude, and direction. Therefore, a landscape image necessary for acquiring a background image by using this service is available. Can be obtained. Here, the shooting point B, the shooting point C, the shooting point T, and the shooting point X in FIG. 5 are set as shooting points, and a landscape image is acquired by designating the direction that is orthogonal to the section and away from the viewpoint 203 as the direction. To do.

  Next, the background image acquisition unit 114 corrects the distortion of the acquired landscape image. Google's Street View service captures and stores landscapes with wide-angle lenses, so the resulting image contains lens distortion. For this reason, in the present embodiment, distortion is removed, and a planar image is generated and used to simplify processing and improve image accuracy. Non-Patent Document 2 is a representative lens distortion removal technique.

  Next, the background image acquisition unit 114 cuts out an image to be used as the background image of the section from the landscape image obtained as a result of distortion removal. On the map, a building behind the section and a space of a road without a building are set as a background image. In the vertical direction, an image from the ground surface to a certain height is cut out. The constant height sets a height generally considered as the height of the building. This height may be set according to the region, and if the map information includes building height information, the highest height of the building corresponding to each section is set based on the map information. Also good. At this time, the coordinates of each vertex of the image are recorded in the image.

  Next, the operation of the fitting composition unit 115 will be described. Non-Patent Document 3 is detailed for converting an image on a plane in a three-dimensional space into a camera coordinate system.

  Non-Patent Document 3 discloses techniques of transformation into a camera coordinate system (section 5.4.2), perspective transformation and projective transformation (section 5.4.3). By combining these, three-dimensional A planar image in space, its position and range can be obtained, and projective transformation to the camera coordinate system is possible when the position, direction and focal length of the camera are obtained. Specifically, each coordinate in the background image is converted into a position on the image sensor of the imaging unit 107 using the following calculation method of Non-Patent Document 3.

  First, a unit vector indicating the direction of the camera is obtained from the direction, the elevation angle, and the tilt angle. A method for obtaining a direction, an elevation angle, and an inclination angle using a triaxial acceleration sensor and a triaxial geomagnetic sensor is known.

  Next, using (Equation 1), a transformation matrix T having a vertical vector component of a unit vector (a, b, c) representing the x, y, and z axis directions of the camera and a camera position vector d is obtained. Ask.

  Next, by using (Equation 2) and (Equation 3), a 3 × 4 transformation matrix representing an inverse transformation with respect to the matrix T, that is, a transformation matrix Tw for converting the world coordinates of the object into the camera coordinate system is obtained. Ask.

  Next, using the above transformation matrix, the world coordinates x, y, z of the object are transformed into coordinate values x ′, y ′, z ′ of the camera coordinate system by (Equation 4).

  Next, using (Equation 5), x ′, y ′, z ′ are positions x ″, y ″, z ″ on the projection plane (imaging device) placed at a distance f from the viewpoint 203. Convert to

  Although the above equation shows the method of coordinate conversion, the above-described coordinate conversion is performed on each pixel in the background image to convert it to a position on the image sensor of the imaging unit 107, and interpolation, averaging, etc. at the converted position are performed. Image processing is performed to generate an image on the image sensor. Since this image processing is general processing, details are not described here.

  Next, the operation of the continuous section determination unit 110 will be described with reference to FIG. Here, the direction of the photographing unit 107 is specifically moved from the direction A toward the direction E, and the section C is set in the section setting unit 112 as a section for displaying the background image.

  When the direction of the photographing unit 107 further changes to the right and becomes the direction of the direction D, the display section switching determination unit 109 sets the straight line to the direction of photographing as the section set in the section setting unit 112 It is detected that the vehicle passes over the end point A, which is one of the end points, and notifies the continuous section determination unit 110.

  Next, the continuous section determination unit 110 extracts another section having the end point A as an end point from the map information. In FIG. 2, section D, section X, and section Y are extracted. Next, the continuous section determination unit 110 selects the one having the smallest angle that intersects the currently set section C from the extracted sections. 2, the section D is parallel to the section C, and the sections X and Y are perpendicular to the section C. Therefore, the section D is selected, and the section setting unit is set as a section for displaying the background image next. 112. The determination rule at this time can be set in more detail and is effective. For example, when the difference between the intersecting angles is within a certain range, a rule for setting a section having a shorter distance from the viewpoint 203 or a section having a far distance may be provided. In addition, by setting a section having a short distance from the viewpoint 203 in the section setting unit 112 regardless of the intersecting angle, a section on a closer side can be selected as the photographing unit 107 is swung left and right. On the contrary, by setting a section with a long distance in the section setting unit 112, the section on the far side can be selected as it is swung left and right. Also, these rules may be implemented as software and hardware and provided as a function that can be set by the user, or the user himself / herself can create and set a new rule by a combination of simple rules. Good.

  Next, the operation of the section changing unit 111 will be described with reference to FIG. Since the section change unit has the first change operation unit 111A and the second change operation unit 111B and each receives a different operation, each operation will be described separately. Here, the direction of the photographing unit 107 is specifically the direction G, and the section C is set in the section setting unit 112 as a section for displaying the background image.

  First, the operation of the first change operation unit 111A will be described. 111 A of 1st change operation parts receive the instruction | indication of the change of the distance from a user. The instruction to change the distance from the user is whether to change the section so that the background image of the section of the farther road is displayed or to change the section to display the background image of the section of the closer road. There are two types of instructions. Upon receiving these instructions, the first change operation unit 111A extracts map information from the map information acquisition unit 102, and extracts information of all road sections therein. Next, 111 A of 1st change operation parts select only the area which has the straight line and the intersection on the direction G among all the areas. In the case of the direction G in FIG. 2, the section E, the section V, the section C, and the section H are selected in order from the side closer to the viewpoint 203. Next, 111 A of 1st change operation parts exclude the thing which the angle which cross | intersects the section C is more than a fixed value from those sections. The user's instruction to the first change operation unit 111A is to change the distance of the section, and is not intended to change the direction, so this operation is performed to prevent the direction from being changed extremely. Here, if it is excluded when the difference in angle is 45 degrees or more, the section V is excluded, and the section E, the section C, and the section H remain. Next, 111 A of 1st change operation parts obtain | require the intersection of each remaining area and the straight line on the direction of the imaging | photography part 107, and sort each area according to the distance between the intersection and the viewpoint 203. FIG. In this case, when the distances are arranged in order of increasing distance, the sections E, C, and H are arranged in this order. Next, in the case where the instruction from the user increases the distance, the first change operation unit 111 </ b> A is a section that is next to the section that is currently set in the section setting section 112 from the sorted sections. Is set in the section setting unit 112. In this case, section H is set. If the instruction from the user is to reduce the distance, a section next to the section currently set in the section setting unit 112 is selected from the sorted sections and set in the section setting unit 112. To do. In this case, section E is set.

  Next, the operation of the second change operation unit 111B will be described. The second change operation unit 111B receives a direction change instruction from the user. There are two types of instructions for changing the direction from the user: instructions to change the section in the clockwise direction or to change the section in the counterclockwise direction. Receiving these instructions, the second change operation unit 111B obtains an intersection between the section extracted from the section setting unit 112 and the direction G. Next, the second change operation unit 111B selects an end point close to the intersection from the end point A and the end point B which are the two end points of the section. In this case, the end point B is selected. Next, the 2nd change operation part 111B takes out map information from the map information acquisition part 102, and extracts the other area which shares the end point B out of it. In this case, section B, section V, and section W are extracted. Next, the second change operation unit 111B obtains angles formed by the extracted sections and the section C that is a section set in the section setting unit 112, and sorts them in the clockwise order. In this case, the sections V, B, and W are arranged in this order. Here, the angle is calculated by taking the direction from the end point B toward the far side.

  Next, when the instruction from the user changes the angle clockwise, the second change operation unit 111B selects the first section from the sorted sections and sets it in the section setting section 112. . In this case, section V is set. When the instruction from the user is to be changed counterclockwise, the last section is selected from the sorted sections and set in the section setting unit 112. In this case, the section W is set.

  According to such a configuration, when the direction of the photographing unit 107 is detected by the direction detection unit, and the direction moves outside the range of the road section set to display the background image, the section is in contact with the section. By automatically selecting the section in the near direction, setting it as the section to display the background image next, acquiring the background image display, and displaying it by fitting and synthesizing with the image other than the background image captured by the shooting unit When displaying the background image of the street by removing only the shielding object in the direction in which the photographing unit is facing when the direction of the photographing unit is changed to the left and right, when the background image of the street is displayed, the section in the continuous street is displayed by the user. Can be selected and displayed without specification.

  The shielding removal display device according to the present invention can continuously select a road section according to the user's intention only by the user's operation of changing the direction of the device. It is useful as a display device for combining and displaying.

DESCRIPTION OF SYMBOLS 101 Position detection part 102 Map information acquisition part 103 Direction detection part 104 Elevation angle detection part 105 Inclination angle detection part 106 Focal length detection part 107 Imaging | photography part 108 Section selection part 109 Display area switching determination part 110 Continuous area determination part 111 Section change part 112 Section setting unit 113 Removal target determination unit 114 Background image acquisition unit 115 Insertion synthesis unit 116 Display unit 201A to 201E Building 202A to 202E Building 203 View point

Claims (7)

Photographing means;
Direction detecting means for detecting the direction of the photographing means;
An elevation angle detection means for detecting an elevation angle of the imaging means;
An inclination angle detecting means for detecting an inclination angle of the photographing means;
A focal length detecting means for detecting a focal length of the photographing means;
Position detecting means for detecting the position;
Map information acquisition means for acquiring map information from the position;
Section selection means for selecting a section for displaying a background image that is a landscape image far from the section of the road from one or a plurality of sections of the road obtained from the direction, the position, and the map information;
A display section switching determination means for determining whether or not to switch a section for displaying the background image from the direction, the position, and the map information;
At the time of switching the section, continuous section determination means for determining and setting a section that is continuous with the section displaying the background image before switching,
Based on the section displaying the background image, removal target determination means for determining a shield to be removed;
Background image acquisition means for acquiring a background image of the shielding object determined to be removed;
A fitting composition that fits and synthesizes the background image of the shielding object and the image of the portion other than the shielding object of the image photographed by the photographing means based on the direction, the elevation angle, the tilt angle, and the focal length. Means,
A shielding object removal display device comprising: a display unit; The said fitting means fits the said background image and the image of parts other than the obstruction | occlusion object of the image image | photographed with the said imaging | photography means by making a position and size correspond. Shield removal display device. The obstruction removal display according to claim 1, wherein the background image acquisition unit acquires a landscape image stored by associating a shooting position and a shooting direction at the time of shooting from a server. apparatus. The shielding object removal display device according to any one of claims 1 to 3, further comprising section changing means for changing a section in which the background image is displayed. The section change means has first change operation means and second change operation means, and operates the first change operation means to change the distance to the section displaying the background image. The shielding object removal display device according to claim 4, wherein the direction of the section in which the background image is displayed is changed by operating the button. 6. The obstruction removal display device according to claim 4, wherein the section changing means displays a map on a screen, and the user instructs a section in which a background image is displayed on the map. . The said background image acquisition means correct | amends distortion of the acquired image, and uses it as a background image, The obstruction removal display apparatus in any one of Claims 1-6 characterized by the above-mentioned.

Images