JP6579706B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to panoramic image processing.

  A technique for obtaining a wider-angle image called a panoramic image by combining a plurality of still images having different line-of-sight directions is known (see Patent Document 1).

JP 2014-155168 A

  A panoramic image is created by setting a projection spherical surface centered on a specific viewpoint and projecting a plurality of images on the inner peripheral surface thereof. At this time, a plurality of images are combined and a panoramic image is obtained by making the adjacent images partially overlap each other. If the viewpoints of the images constituting the panoramic image are coincident, in principle there is no discontinuity in the connection between the images or image distortion. However, the viewpoints of a plurality of images to be combined do not always match. For example, in the case of an omnidirectional camera equipped with a plurality of cameras, the viewpoint position of each camera cannot be physically matched. There is distortion in the case of.

  In such a background, an object of the present invention is to solve a problem caused by a difference in viewpoint between different cameras in a technique for obtaining a panoramic image by combining a plurality of images.

According to the first aspect of the present invention, there is provided an image data receiving unit that receives image data of a first still image and image data of a second still image in which overlapping objects are captured from different viewpoints, and the first still image, A composite image forming unit that projects the second still image onto a projection sphere to form a composite image, a designation receiving unit that accepts designation of a specific position in the composite image, and the designation accepting unit A selection unit that selects one of the first still image and the second still image including a designated position, and a viewpoint of the first still image, a viewpoint of the second still image, and the projection sphere The centers of the image processing apparatuses are located at different positions .

  According to the first aspect of the present invention, when a specific part is designated in a state where the composite image (panoramic image) is displayed, a single photograph image including the part (a plurality of still images constituting the panoramic image) is displayed. One of the images) is selected. Since the single photograph image does not have a problem of joints in the panoramic image, the problem of misalignment in the panoramic image is solved. In claim 1, two still images are targeted. However, claim 1 can be applied to two images included in a plurality of still images. Therefore, the target still image may be three or more.

  According to a second aspect of the present invention, in the first aspect of the invention, the selection unit is configured to determine the specific position of the first still image and the second still image as viewed from the center of the projection sphere. One having an image orientation close to the direction is selected. According to the second aspect of the present invention, a still image (single photograph image) in which the specified specific position is closer to the center of the screen is selected. The orientation of the image is defined as a direction perpendicular to the still image screen. As an equivalent definition, an extension direction of a line connecting the viewpoint of the still image and the screen center of the still image can be defined as the orientation of the image.

  A third aspect of the present invention includes the display control unit according to the first or second aspect of the present invention, further comprising a display control unit that performs control to display the first still image or the second still image on a display device. One still image and the second still image have an overlapping portion, and the display control unit specifies a specific position in the overlapping portion in a situation where the first still image is displayed. If the designated position is closer to the center of the second still image than the center of the first still image, the second still image is displayed.

  According to the third aspect of the present invention, switching of a still image (single photograph image) accompanying a change in line of sight is performed. At that time, a still image having a focus point at the center is selected.

According to a fourth aspect of the present invention, there is provided an image data receiving step of receiving image data of a first still image and image data of a second still image obtained by capturing an overlapping object from different viewpoints; and the first still image, The composite image forming step of forming the composite image by projecting the second still image onto the projection sphere, the specification receiving step for receiving the specification of a specific position in the composite image, and the specification receiving step A selection step of selecting one of the first still image or the second still image including a designated position, and a viewpoint of the first still image, a viewpoint of the second still image, and the projection sphere Are the image processing methods at different positions .

According to a fifth aspect of the present invention, there is provided a program for causing a computer to read and execute the image data of the first still image and the image data of the second still image obtained by capturing images of overlapping objects from different viewpoints. A receiving image data receiving step, a composite image forming step of forming the composite image by projecting the first still image and the second still image onto a projection sphere, and designation of a specific position in the composite image a designation accepting step of accepting, either to execute a selecting step of selecting the first still image and the second still image including the designated position accepted by the designation accepting step, the first still image The viewpoint, the viewpoint of the second still image, and the center of the projection sphere are image processing programs at different positions .

  According to the present invention, it is possible to solve a problem caused by a difference in viewpoint between different cameras in a technique for obtaining a panoramic image by combining a plurality of images.

It is a principle diagram of a panoramic image. It is a principle figure which shows the principle of the switch from a panoramic image to a single photograph image. It is a block diagram of an image processing device. It is a drawing substitute photograph showing an example of a composite image in which a panoramic image and point cloud data are superimposed. It is a flowchart which shows an example of the procedure of a process. It is a principle figure which shows the principle of switching of a single photograph image. It is a principle figure which shows the principle of switching of a single photograph image. It is a principle figure which shows the principle of switching of a single photograph image. It is a principle figure which shows the change of a display image when the eyes | visual_axis in a panoramic image is moved. It is a drawing substitute photograph which shows an example of the display image which changed display magnification.

1. First embodiment (outline)
First, a problem that occurs when a plurality of images with different viewpoints are combined will be described. FIG. 1 shows the principle in the case where three still images partially overlapping using three cameras with different positions (viewpoints) are taken and projected onto the inner peripheral surface of a projection sphere to create a panoramic image. It is shown. Here, the still image that is the basis of the panoramic image is referred to as a single photograph image. A single photo image is a still image taken by each camera, and a panoramic image can be obtained by combining a plurality of single photo images. For example, in the case of FIG. 1, there are three cameras, three single photograph images are taken, and a panoramic image is obtained by combining the three single photograph images. The projection sphere is virtually set, and a projection image is formed based on the setting that there is actually a spherical projection surface.

  In the case of FIG. 1, the viewpoint of the panoramic image is the center of the projection sphere, but the viewpoints of the three images constituting the panoramic image do not coincide with the center of the projection sphere. Therefore, when viewed strictly, image distortion occurs. In addition, image misalignment occurs in the overlapping portion of the two images. When a panoramic image is used as an image for obtaining a general view of the surroundings, the above phenomenon is not a big problem. However, when a work such as measurement using an image or creation of a drawing based on the image is performed, the above phenomenon is a problem. It becomes.

  Therefore, as a method for solving the above problem, when a specific point or region in the panoramic image is specified, a single photograph including the specified point or region (in the plural still images constituting the panoramic image). And switch the display to the single photograph image. In this case, the viewpoint is changed from the viewpoint of the panoramic image (the center of the projection sphere) to the viewpoint at which the single photograph image is taken. A single photograph is free from problems such as image distortion and misalignment associated with image synthesis for obtaining the panoramic image described above, so that an image that can be used for measurement or the like can be obtained.

  FIG. 2 shows a state where the panoramic image of FIG. 1 is switched to a single photograph image. In this case, the direction of the line of sight in FIG. 1 is detected, and a single photograph image having the line of sight (orientation) closest to that direction is selected. FIG. 2 shows a state in which the viewpoint is changed from C0 to C1 and the panorama image is changed (switched) to the single photograph image.

(Hardware configuration)
FIG. 3 shows a block diagram of the embodiment. FIG. 3 shows an image processing apparatus 100, an all-around camera 200, a laser scanner 300, and a display apparatus 400.

  The image processing apparatus 100 functions as a computer and has a function unit described later. The omnidirectional camera 200 is a multi-lens camera for omnidirectional photography, and shoots 360 degrees above and around. In this example, the all-around camera 200 includes six cameras. Of the six cameras, five cameras are disposed at equiangular angular positions (every 72 °) as viewed from above in the horizontal direction. The remaining one camera is directed vertically upward (elevation angle 90 °). These six cameras are set so that the angles of view (shooting ranges) partially overlap. A panoramic image is obtained by combining still images obtained from the six cameras.

  In the all-round camera 200, the relative positional relationship between the orientation and position of each camera is examined in advance and is known. Also, due to physical problems, the position of the viewpoint (projection center) of each camera does not match. The all-round camera is described in, for example, Japanese Patent Application Laid-Open Nos. 2012-204982 and 2014-71860. A commercially available product can also be used as the all-round camera 200. Examples of commercially available all-around cameras include the product name Ladybug 3 manufactured by Point Gray. It is also possible to use a form in which a plurality of still images having different shooting directions are obtained using a camera having a rotation mechanism instead of the all-around camera, and a panoramic image is obtained by combining the plurality of still images. Of course, the panoramic image is not limited to the all-round image, and may be in a specific angle range. Image data of a plurality of still images taken from different directions from the omnidirectional camera 200 is sent to the image processing apparatus 100.

  The six cameras capture still images at a specific timing. Photographing can also be performed at specific time intervals. For example, it is also possible to obtain an all-round image by sequentially operating six cameras with a time difference and synthesizing the obtained images. Moreover, the form which image | photographs a moving image is also possible. When shooting a moving image, frame images constituting the moving image (for example, frame images captured 30 images per second) are handled as still images.

  The laser scanner 300 irradiates the object with laser light and detects the reflected light, thereby detecting the direction and distance to the object. The laser scanner 300 performs laser scanning in the same range as the shooting range of the all-around camera 200 while moving the laser irradiation unit and the reflected light receiving unit up and down and left and right. The laser scanner is described in Japanese Patent Application Laid-Open Nos. 2008-268004 and 2010-151682.

  The positional relationship and orientation relationship between the laser scanner 300 and the all-around camera 200 are acquired in advance and are known. The coordinate system of the point cloud position data acquired by the laser scanner 300 may be an absolute coordinate system or a relative coordinate system. The absolute coordinate system is a coordinate system that describes a position measured using GNSS or the like. The relative coordinate system is a coordinate system described with the machine center of the all-round camera 200 and other appropriate positions as the origin.

  The display device 400 is an image display device such as a liquid crystal display. As the display device 400, a display of a tablet or a personal computer can be used. Image data processed by the image processing apparatus 100 is sent to the display device 400, and the image is displayed.

  FIG. 3 shows functional units included in the image processing apparatus 100. The image processing apparatus 100 includes a CPU, various storage devices such as an electronic memory and a hard disk device, various arithmetic circuits, and an interface circuit, and has a function as a computer that executes functions to be described later. The image processing apparatus 100 includes an image data receiving unit 101, a panoramic image forming unit 102, a designation receiving unit 103, a line-of-sight detection unit 104, a single photograph image selection unit 105, a display image control unit 106, a point cloud position data acquisition unit 107, an image And a point cloud image synthesis unit 108 and a three-dimensional position calculation unit 109.

  These functional units may have a software configuration (a configuration realized by executing a program by a CPU) or may be configured by a dedicated circuit. Moreover, the function part comprised by software and the function part comprised by the circuit for exclusive use may be mixed. For example, each functional unit shown in the figure is composed of one or more combinations of electronic circuits such as a PLD (Programmable Logic Device) such as a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array). It is configured.

  Whether each functional unit constituting the image processing apparatus 100 is configured by dedicated hardware or software by executing a program in the CPU is determined in consideration of required calculation speed, cost, power consumption, and the like. Is done. For example, if a specific functional unit is configured with FPGA, it is advantageous in terms of processing speed but is expensive. On the other hand, a configuration in which a specific functional unit is realized by executing a program by the CPU can save hardware resources, and is advantageous in terms of cost. However, when the functional unit is realized by the CPU, the processing speed is inferior to that of dedicated hardware. Further, when the functional unit is realized by the CPU, it may not be able to cope with complicated calculations. It should be noted that the configuration of the functional unit with dedicated hardware and the configuration with software are equivalent from the viewpoint of realizing a specific function, although there are the differences described above.

  Hereinafter, each functional unit included in the image processing apparatus 100 will be described. The image data receiving unit 101 receives image data of a still image taken by the omnidirectional camera 200. Specifically, the image data receiving unit 101 receives image data of still images taken by six cameras included in the all-around camera 200. Each of the still images taken by these six cameras becomes a single photograph image.

  The panorama image forming unit 102 sets a projection sphere, and projects six still images (single photograph images) received by the image data receiving unit 101 on the inner peripheral surface of the projection sphere. By projecting onto the projection sphere, a panoramic image obtained by synthesizing six still images is obtained. This panoramic image is viewed from the center of the projection sphere. A predetermined specific value or infinity is adopted as the radius of the projection sphere. As the position of the center of the projection sphere, for example, the position of the mechanical center of gravity of the all-round camera 200 and other easily manageable positions are employed.

  The designation receiving unit 103 receives the designation of the point of interest in the composite image (panoramic image) formed by the panoramic image forming unit 102. For example, consider a case where there is a panoramic image obtained by synthesizing two still images taken of overlapping objects, and the panoramic image is displayed on a PC (personal computer) screen. In this case, the user operates a GUI (graphical user interface) of the PC to designate a point to be enlarged and displayed as an attention point. The designation accepting unit 103 accepts data related to the operation content.

  The line-of-sight detection unit 104 detects the line of sight from the center of the projection sphere (the viewpoint of the panoramic image) to the point of interest received by the designation receiving unit 103. In this example, a vector from the center of the projection sphere to the point of interest is calculated. Since the relative relationship of the external orientation elements of the cameras constituting the all-round camera 200 is known in advance, a vector from the center of the projection sphere to the point of interest designated on the inner circumferential surface of the projection sphere is calculated. Can do. Specifically, first, the coordinates of the point of interest on the projection sphere are acquired. Next, a vector connecting the center of the projection sphere and the acquired coordinate position of the target point is set. This vector is the line of sight to the point of interest in the panoramic image.

  The single photograph image selection unit 105 selects a single photograph image having a line of sight closest to the line of sight detected by the line of sight detection unit 104. The line of sight of the single photograph image is acquired as a direction perpendicular to the screen of the single photograph image. The line of sight of the single photograph image may be acquired as a direction from the viewpoint of the single photograph image toward the center of the angle of view of the single photograph image (the center of the field of view).

  The single photograph image selection unit 105 performs the following processing. First, the direction of the line of sight of each of a plurality of single photograph images is acquired. Next, the direction of the line of sight to the point of interest in the panoramic image is compared with the direction of the line of sight to the point of interest in the single photograph image, and a single photograph image having the line of sight closest to the direction of the line of sight to the point of interest is obtained. select. Although the viewpoint of the panoramic image is different from the viewpoint of the single-photo image, the difference is small compared to the distance to the point of interest. Therefore, the above processing causes the point of interest to be as close to the center of the angle of view as possible. A photographic image is selected.

  The display image control unit 106 performs control for causing the display device 400 to display one or both of the panoramic image created by the panoramic image forming unit 102 and the single photo image selected by the single photo image selection unit 105. As a form of image display, there are a form of displaying a panoramic image and a single photograph image which is selected by the user, and a form of dividing both screens and displaying both. The display image control unit 106 performs control related to switching of single photograph images.

  The point cloud position data acquisition unit 107 acquires 3D point cloud position data measured by the laser scanner 300. The image and point cloud image combining unit 108 generates an image (image and point cloud combined image) by combining the panoramic image and the three-dimensional point cloud position data.

  In the point cloud position data, since the direction of each point constituting the point cloud viewed from the laser scanner 300 is known, when each point viewed from the laser scanner 300 is projected onto the inner peripheral surface of the projection sphere, the projected point is defined as a pixel. A point cloud image (two-dimensional image composed of point clouds) can be created. This point cloud image is an image composed of points and can be handled in the same manner as a normal still image.

  Here, the relative positional relationship and the relative orientation relationship between the all-round camera 200 and the laser scanner 300 are acquired in advance and known. Therefore, the still image photographed by the camera of the omnidirectional camera 200 and the above point cloud image can be superimposed by the same method as the method of synthesizing the images of the six cameras constituting the omnidirectional camera 200. Based on this principle, a panoramic image obtained by combining a plurality of still images captured by the all-around camera 200 and the above point cloud image can be superimposed, and a panoramic image obtained from the image captured by the camera and the point cloud A composite image obtained by superimposing the data is obtained. This processing is performed in the image and point cloud image combining unit 108. An example of a composite image obtained by superimposing an image and a point cloud image is shown in FIG.

  The three-dimensional position acquisition unit 109 acquires the three-dimensional position of the point (point of interest) designated by the designation receiving unit 103 based on the point cloud position data. Specifically, the point of the point cloud position data corresponding to the position on the screen of the point of interest designated by the designation receiving unit 103 is acquired from the combined image of the image and the point cloud illustrated in FIG. The three-dimensional coordinate position of the acquired point is acquired from the point group position data acquired by the point group position data acquisition unit 107. If there is no point corresponding to the point of interest, (1) a nearby point is selected and the three-dimensional position of that point is acquired, (2) a plurality of neighboring points are selected, and the average of the three-dimensional positions is selected. Get the value, (3) Get the value of the 3D coordinates of the point of interest by selecting a plurality of nearby points, then selecting the points that are close to the 3D position, and obtaining the average value To do.

(Example of processing)
Hereinafter, an example of a procedure of processing executed by the image processing apparatus 100 in FIG. 3 will be described. A program for executing the processing described below is stored in a storage area in the image processing apparatus 100 or an appropriate external storage medium, and is executed by the image processing apparatus 100.

  FIG. 5 is a flowchart illustrating an example of a processing procedure. When the processing is started, image data taken by the all-around camera 200 is accepted (step S101). In this process, image data of six single photograph images photographed by the six cameras included in the all-round camera 200 is acquired. This process is performed in the image data receiving unit 101.

  Next, a panoramic image is created by synthesizing six single photograph images (step S102). In this process, a projection sphere is set, and each single photograph image is projected onto the inner peripheral surface thereof to create a panoramic image. This process is performed in the panoramic image forming unit 102. When the panorama image is created, the image data is sent to the display device 400, and the panorama image is displayed on the display device 400 (step S103).

  When the user designates a specific point on the panoramic image as a point of interest while the panoramic image is displayed on the display device 400, the determination in step S104 is YES, and the process proceeds to step S105. If the user does not specify the point of interest, the panoramic image displayed in step S103 is continuously displayed.

  In step S105, the direction of the line of sight to the point of interest received by the designation receiving unit 104 (the direction from the center of the projection sphere to the point of interest) is detected (step S105). This process is performed in the line-of-sight detection unit 104. When the direction of the line of sight to the point of interest in the panoramic image is detected, a single photograph image having the line of sight closest to the direction of the line of sight is selected (step S106).

  In this process, the direction from the center of the projection sphere to the point of interest designated on the panoramic image is specified as the first direction. Then, the direction from the viewpoint of each single photograph image to the center of the screen, in other words, the direction perpendicular to the screen of the single photograph image is acquired as the second direction. Here, since there are a plurality of (6 in this case) single photograph images, the second direction is selected from among the single photograph images having the second direction closest to the first direction. .

  Next, the single photograph image selected in step S106 is displayed (step S107). By this processing, switching from the panoramic image displayed in step S103 to the single photograph image selected in step S106 is performed. Next, it is determined whether or not an instruction for switching to a panoramic image has been issued from the user (step S108). If there is an instruction to switch to a panoramic image, the process is executed again from step S103 onward. If so, the process proceeds to step S109.

  When the designated position of the point of interest is changed by the user on the single photograph image screen displayed in step S107 and the designated position approaches the end of the display screen, the determination in step S109 is YES, and this is displayed at this time. The display is switched to the single photo image adjacent to the single photo image being displayed (step S110). In this processing, first, it is determined whether or not the point of interest in the single photograph image (single photograph image 1) displayed at that time is in a portion overlapping with the adjacent single photograph image (single photograph image 2). . When the focus point is in the overlap portion, it is determined whether the focus point and the center of the screen are close to the single photograph image 1 or the single photograph image 2, and the distance between the focus point and the center of the screen is determined. When the single photograph image 2 is close to, the determination in step S109 is YES, and the display is switched to the single photograph image 2 (step S110). If the single photograph image 1 is close to the point of interest from the screen center, the process returns to the stage before step S108, and the single photograph image displayed in step S107 is displayed as it is.

  Hereinafter, an example of the processing from step S109 to step S110 will be described with reference to the drawings. FIG. 6 shows a state in which a single photograph image taken by the camera 1 is displayed. FIG. 7 shows a state in which the position of the point of interest has moved in the clockwise direction in FIG. 6 and the point of interest has moved to the overlapping portion of the shooting range of the camera 1 and the camera 2 in FIG.

  In the state of FIG. 7, since the point of interest is close to the center of the shooting range of the camera 1, the single photograph image taken by the camera 1 is still displayed. FIG. 8 shows a state where the point of interest further moves in the clockwise direction from the state of FIG. In the case of FIG. 8, the point of interest is closer to the center of the angle of view (shooting range) of the camera 2 than the center of the shooting range of the camera 1, so that the display image is switched to a single photograph image taken by the camera 2. In this case, with the movement of the point of interest, the image displayed on the display device 400 is switched from the single photo image taken by the camera 1 to the single photo image taken by the camera 2. In this case, the position of the viewpoint is changed corresponding to the switching of images.

(Change of display range in panorama display)
It is not realistic to display the entire panoramic image on the display device 400, and usually a part of the panoramic image is displayed on the display device 400. FIG. 9 shows an example of such a case. In this case, the point of interest is set at the center of the displayed screen. When the display range is changed, the point of interest moves on the spherical surface, and the displayed range is changed. In the example of FIG. 9, the range displayed on the panoramic image moves. In this case, the viewpoint position does not change.

(Example when the display magnification is changed)
FIG. 10A shows an example of a low-magnification panoramic display. In FIG. 10A, an image shift that becomes a problem in the panorama display appears. FIG. 10B shows a case where the display magnification is increased in the state of FIG. In this case as well, the image shift that is a problem in the panorama display appears. FIG. 10 (C) shows a case where, in the panoramic display of FIG. 10 (B), attention is paid to the vicinity of the center, switching to single photograph display including the point of interest, and enlargement of the image. As can be seen from FIG. 10C, by switching from the panoramic image to the single photo display, the image shift that is a problem in the panoramic image is eliminated. Note that the entire image in FIG. 10C is blurred because of camera performance, and a clearer image can be obtained by using a camera with higher resolution.

(Other)
As another method for accepting designation of a point of interest, there is a method of displaying a panoramic image on a touch panel display and accepting designation of the point of interest by touching the display such as a touch pen. As another method for receiving the designation of the point of interest, the line of sight of the user who is viewing the panoramic image is detected, the intersection of the line of sight and the image plane of the panoramic image is detected, and the position is accepted as the designated position. A method is mentioned. The technique for detecting the line of sight is described in, for example, Japanese Patent Laid-Open No. 2015-118579.

Claims (5)

An image data receiving unit that receives the image data of the first still image and the image data of the second still image obtained by capturing an overlapping target from different viewpoints;
A composite image forming unit that projects the first still image and the second still image onto a projection sphere to form a composite image;
A designation accepting unit that accepts designation of a specific position in the composite image;
A selection unit that selects one of the first still image and the second still image including the designated position received by the designation receiving unit;
The image processing device in which the viewpoint of the first still image, the viewpoint of the second still image, and the center of the projection sphere are at different positions .   The selection unit selects a direction having an image orientation close to a direction of the specific position viewed from a center of the projection sphere in the first still image and the second still image. Item 8. The image processing apparatus according to Item 1. A display control unit that performs control to display the first still image or the second still image on a display device;
The first still image and the second still image have an overlapping portion,
The display control unit
In the situation where the first still image is displayed,
When a specific position in the overlapping portion is specified, the second still image is determined when the specified position is closer to the center of the second still image than the center of the first still image. The image processing apparatus according to claim 1, wherein the image processing apparatus is displayed. An image data accepting step for accepting image data of a first still image and image data of a second still image obtained by imaging an overlapping target from different viewpoints;
A composite image forming step of projecting the first still image and the second still image onto a projection sphere to form a composite image;
A designation accepting step for accepting designation of a specific position in the composite image;
A selection step of selecting one of the first still image or the second still image including the designated position received in the designation receiving step,
An image processing method in which the viewpoint of the first still image, the viewpoint of the second still image, and the center of the projection sphere are at different positions . A program that is read and executed by a computer,
An image data accepting step for accepting image data of a first still image and image data of a second still image obtained by imaging an overlapping target from different viewpoints;
A composite image forming step of projecting the first still image and the second still image onto a projection sphere to form a composite image;
A designation accepting step for accepting designation of a specific position in the composite image;
Selecting one of the first still image and the second still image including the designated position accepted in the designation accepting step; and
An image processing program in which the viewpoint of the first still image, the viewpoint of the second still image, and the center of the projection sphere are at different positions .

Images