JP2013027021A - Omnidirectional imaging device and omnidirectional imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable real time displaying of azimuth information for a bearing displayed simultaneously with a converted image when converting an omnidirectional image into a panoramic image or a planar perspective projection image.SOLUTION: An omnidirectional imaging device has: omnidirectional imaging means for imaging an omnidirectional image; azimuth acquisition means for acquiring azimuth information; and control means for controlling operations of the whole device. The control means acquires by the azimuth acquisition means azimuth information for a certain direction to be a reference in a screen of an omnidirectional images photographed by the omnidirectional imaging means, and adds the acquired azimuth information to omnidirectional image information as meta data. Thereby, the azimuth information can be easily acquired from the photographed omnidirectional image to an image obtained by performing panoramic expansion or planar perspective projection conversion.

Description

  The present invention relates to an omnidirectional imaging apparatus and an omnidirectional imaging method, and more particularly to a technique suitable for use in omnidirectional imaging in which an omnidirectional image, that is, a field of view of 360 degrees around the entire circumference is photographed at once.

  An omnidirectional camera equipped with an omnidirectional mirror or an omnidirectional fisheye lens is an imaging device that captures a landscape of the entire periphery, and has conventionally been considered for various uses such as surveillance cameras and robot navigation. A single camera captures 360-degree circular or circular omnidirectional images. Then, by performing a geometric conversion process on the photographed omnidirectional image, it is converted into a panoramic developed image or a planar perspective projection image, and a desired image in all directions can be obtained. Japanese Patent Application Laid-Open No. 2004-133867 discloses a technique for partially cutting out and distributing a video from a wide viewing angle camera such as an omnidirectional camera.

  Furthermore, there is a demand for acquiring azimuth information about a certain subject with respect to an omnidirectional image photographed by the omnidirectional mirror or an omnidirectional camera equipped with an omnidirectional fisheye lens. In Patent Document 2, an omnidirectional camera is used to capture an image of a magnetic stylus together with a landscape, and the direction of a subject on the landscape is calculated from an angle with the direction of the magnetic needle to detect azimuth information. ing.

JP 2006-115046 A Japanese Patent Laid-Open No. 2004-117696

  As described above, in the omnidirectional imaging apparatus described in Patent Document 2, an omnidirectional camera equipped with an omnidirectional mirror or an omnidirectional fisheye lens is used to capture an image of a magnetic stylus together with the landscape, The orientation information is detected by calculating the orientation of the subject from the angle with the direction of the magnetic needle. However, in that case, it is not easy to obtain azimuth information for an image obtained by panoramic development or plane perspective projection conversion from the captured omnidirectional image.

Further, when the panorama unfolded image or the plane perspective projection image display portion is changed by operation with respect to the omnidirectional image, there is a problem that it is difficult to deal with the display of the azimuth information at that time in real time.
In view of the above-described problems, the present invention displays real-time azimuth information displayed simultaneously with a converted image when converting a captured omnidirectional image into a panoramic developed image or a planar perspective projection image. The purpose is to be able to do this.

  An omnidirectional imaging apparatus according to the present invention includes an omnidirectional imaging unit having a unit that captures an image in all directions, an azimuth acquisition unit that acquires azimuth information, and a control unit that controls the operation of the entire apparatus. The means obtains azimuth information about a certain reference direction of the screen of the omnidirectional image photographed by the omnidirectional imaging means by the azimuth obtaining means and attaches it to the omnidirectional image as metadata. Features.

  According to the present invention, azimuth information in the upper direction of an image is attached as metadata to an omnidirectional image taken by an omnidirectional camera equipped with an omnidirectional mirror or an omnidirectional fisheye lens. Then, when the captured omnidirectional image is converted into a panoramic developed image or a planar perspective projection image, the azimuth information of the displayed direction is obtained from the azimuth information metadata attached to the omnidirectional image. Calculated. Thereby, it is possible to display the azimuth information in the direction displaying simultaneously with the converted image in real time.

It is a block diagram which shows the structure of the omnidirectional imaging device which is embodiment of this invention. It is the figure which showed the external appearance structure of the omnidirectional imaging device which is embodiment of this invention. It is the figure which showed the example of a screen display of the all-around circular image of an omnidirectional imaging device. It is the figure which showed the example of a screen display of the perspective projection image of an omnidirectional imaging device. It is the figure which showed the example of a screen display of the all-around circular image of an omnidirectional imaging device. It is the figure which showed the example of a screen display of the panoramic expansion image of an omnidirectional imaging device. It is the figure which showed the example of a screen display of the panoramic expansion image of an omnidirectional imaging device.

Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.
(First embodiment)
FIG. 1 is a block diagram showing a configuration example of an omnidirectional camera 100 equipped with an omnidirectional mirror as an embodiment of an omnidirectional imaging apparatus to which the present invention is applied, and FIG. 2 is equipped with an omnidirectional mirror. 1 is an external configuration diagram illustrating an embodiment of an omnidirectional camera 100. FIG.

  In FIG. 1, the omnidirectional camera 100 includes a control unit 120 that controls the operation of the entire apparatus in order to control each function for capturing an omnidirectional image. The control unit 120 includes a CPU, a ROM, a RAM, and the like, and realizes operation control of each unit described below by the CPU executing a program stored in the ROM.

  1 and 2, the omnidirectional mirror 101 causes the imaging sensor unit 102 to capture a 360-degree all-around scene by specular reflection using a hyperboloidal mirror. The omnidirectional mirror 101 is not limited to a hyperboloidal mirror, and may be a mirror that reflects a 360-degree all-around landscape by a spherical mirror, or an all-around fisheye lens for imaging the all-around landscape, for example. .

  The imaging sensor unit 102 is an imaging element such as a CMOS that converts a light image formed on the imaging surface into a digital electric signal by photoelectric conversion. The azimuth sensor unit 103 is a sensor provided for performing azimuth acquisition for acquiring azimuth information. For example, it is a magnetoresistive effect element that detects the geomagnetism and acquires the magnetic field strength, and sends the measured result to the direction information processing unit 106.

  The development processing unit 104 performs predetermined pixel interpolation and color conversion processing on the digital electrical signal obtained by photoelectric conversion from the imaging sensor unit 102, and generates a digital image such as RGB or YUV. In addition, the development processing unit 104 performs predetermined arithmetic processing using the developed digital image, and performs image processing such as white balance, sharpness, contrast, and color conversion based on the obtained arithmetic result. Yes.

  The image geometric conversion processing unit 105 performs image geometric conversion processing on the digital image captured by the imaging sensor unit 102 and subjected to development processing and image processing by the development processing unit 104. When the imaging sensor unit 102 captures an image with all pixels, the image is captured as an all-around circular image by capturing the entire landscape displayed on the mirror surface of the omnidirectional mirror 101. In this case, a panoramic expansion conversion process for converting the captured all-around circular image into a cylindrical model, or a plane perspective projection conversion process for converting to a virtual plane without distortion of a partially cut-out screen of the all-around ring image. To generate a panoramic development image or a plane perspective projection conversion image.

  When the imaging sensor unit 102 captures an image by partially reading out the total number of pixels, it captures a part of the entire surrounding subject reflected on the mirror surface of the omnidirectional mirror 101, a cut-out screen, The distortion correction of the image of the part is performed.

  The azimuth information processing unit 106 is a so-called electronic compass that electronically outputs azimuth information to the image geometric transformation processing unit 105 from the result measured by the azimuth sensor unit 103. Further, when the display position of the image is changed by the operation from the operation unit 108 with respect to the panorama development image or the perspective projection conversion image geometrically converted by the image geometric conversion processing unit 105, It calculates from the orientation information of the ring image and outputs the corresponding orientation information.

  The drawing display processing unit 107 performs processing necessary for displaying on the display device 110 the digital image captured by the imaging sensor unit 102 and subjected to development processing and image processing by the development processing unit 104. Specifically, the image geometric conversion processing unit 105 performs appropriate processing in display according to an instruction from the operation unit 108, such as enlarging or reducing each of the geometrically transformed panorama developed image or planar perspective projection converted image.

  The operation unit 108 gives an instruction as to which part of the photographed full-circle image is to be cut out and converted into a panoramic developed image to be displayed as a panoramic developed image. Alternatively, an instruction is given as to which range is to be displayed as a plane perspective projection conversion image when converting to a plane perspective projection image. For example, a keyboard or a mouse is used as the operation input means.

  The image recording unit 109 records the all-around circular image captured by the imaging sensor unit 102 and subjected to the development processing and the image processing by the development processing unit 104 together with the direction information output from the direction information processing unit 106. Then, the azimuth information recorded together with the recorded all-round image is output to the image geometric transformation processing unit 105 by the operation from the operation unit 108, and the image geometric transformation processing is performed. As a result of performing the image geometric conversion process, the image information is output to the drawing display processing unit 107 together with the corresponding direction information calculated by the direction information processing unit 106.

  The display device 110 displays the panorama development image or the plane perspective projection conversion image geometrically converted by the image geometric conversion processing unit 105 together with the graphic drawing of the azimuth information generated by the drawing display processing unit 107.

Next, details of the operation of the omnidirectional camera 100 which is an embodiment of the omnidirectional imaging apparatus of the present invention will be described.
First, the captured image of the entire circular ring is subjected to plane perspective projection conversion to display the image, or the recorded circular image of the entire circle is subjected to plane perspective projection conversion to display the image. The operation of displaying is described.

  When the photographing operation is started, the entire surrounding landscape reflected on the mirror surface of the omnidirectional mirror 101 is imaged by the imaging sensor unit 102 and becomes a digital image subjected to development processing and image processing by the development processing unit 104. Photographed as a surrounding ring image. The captured omnidirectional annular image is sent to the image geometric transformation processing unit 105, subjected to plane perspective projection transformation processing, and is obtained as a plane perspective that becomes a virtual plane without distortion of a screen obtained by partially cutting out the omnidirectional annular image. A projection conversion image is generated.

  At the same time, in parallel with the start of the photographing operation, the direction sensor unit 103 measures the direction of the image sensor unit 102 in a certain direction, for example, the upper direction, and outputs the result to the direction information processing unit 106. The azimuth information processing unit 106 outputs the azimuth information to the image geometric transformation processing unit 105 as data such as NMEA-0183 format based on the result. The azimuth information output from the azimuth information processing unit 106 may be an azimuth other than the upper direction of the all-around circular image as long as the direction as a reference is uniquely determined. The azimuth information output from the azimuth information processing unit 106 can be recorded as metadata by the image recording unit 109 together with the captured all-around circular image.

  For example, the video is recorded in a format that conforms to the AVCHD standard, and the azimuth information is constant in the area for storing user data in the AVCHD format. Keep a record every hour. The image geometric transformation processing unit 105, which has received the all-round image from the development processing unit 104 and the orientation information from the orientation information processing unit 106, first outputs the data to the drawing display processing unit 107 as they are.

  Similarly, the all-round annular image recorded by the image recording unit 109 is output to the drawing display processing unit 107 as it is together with the orientation information recorded together. The drawing display processing unit 107 to which the omnidirectional annular image and the azimuth information are input draws so that the direction in which the omnidirectional annular image is displayed matches the azimuth information of the direction, and the display device 110. To output and display. For example, the display is performed by drawing so as to indicate a certain direction in the upper direction of the displayed all-round image.

  FIG. 3 shows a screen display example of the all-around circular image at this time. The user who sees the screen display designates a certain part of the all-around ring image as shown in FIG. 3 from the operation unit 108 to cut out a part of the all-around ring image and perform the perspective projection conversion. Is sent to the image geometric transformation processing unit 105.

  Upon receiving the instruction, the image geometric transformation processing unit 105 requests the azimuth information processing unit 106 to receive azimuth information related to the direction of the central portion of the area of the corresponding all-round image. Then, the image geometric transformation processing unit 105 performs plane perspective projection transformation on a certain area of the all-round annular image, and draws and displays a plane perspective projection transformed image together with the direction information obtained from the direction information processing unit 106. Output to the unit 107. The drawing display processing unit 107, to which the plane perspective projection conversion image and the azimuth information are input, draws so that the direction of the display screen can be understood, and outputs it to the display device 110 for display.

  FIG. 4 shows a screen display example of the plane perspective projection conversion image and orientation information at this time. Further, both the all-round image in FIG. 3 and the plane perspective projection conversion image in FIG. 4 are simultaneously displayed on the display device 110. Accordingly, when the user changes a part of the all-around annular image cut out by the operation unit 108, the display of the plane perspective projection conversion image and the azimuth information can be changed in real time.

  Further, the displayed planar perspective projection conversion image is recorded in the image recording unit 109 as a still image. At the time of this recording, the orientation information is recorded in the GPS Img Direction item of the GPS Info Tag portion of the EXIF attached information together with the plane perspective projection conversion image. As a result, when the plane perspective projection conversion image recorded later is displayed, this data can be used to display the azimuth information.

  Next, the panoramic expansion conversion process is performed on the photographed all-round annular image and the image is displayed. Or the operation | movement which performs a panoramic expansion conversion process with respect to the recorded all-around circular image, and displays the image is demonstrated. The shooting operation is started, and the imaged all-around circular image is sent to the image geometric transformation processing unit 105, panorama development conversion processing is performed, and the panorama development image obtained by opening the whole-circle image as a cylinder is obtained. Generated.

  Operation taken by the azimuth sensor unit 103 and the azimuth information obtained from the azimuth information processing unit 106 to be output to the drawing display processing unit 107 and displayed on the display device 110 in parallel with the captured all-round annular image Is the same as in the case of the above-described plane perspective projection conversion processing. The same is true for the operation of displaying the all-round annular image recorded in the image recording unit 109.

  The user who views the screen display of the all-around ring image displayed on the display device 110 designates a certain portion of the all-around ring image as shown in FIG. Then, the operation unit 108 sends an instruction to the image geometric conversion processing unit 105 to cut out from a certain part of the all-around circular image and perform panorama development conversion processing.

  Upon receiving the instruction, the image geometric transformation processing unit 105 requests the azimuth information processing unit 106 to receive azimuth information related to the direction of the designated position of the corresponding all-round image. Then, the image geometric transformation processing unit 105 performs panorama development conversion processing that is opened from a specified position of the corresponding all-round image, and draws a panoramic development image together with the direction information obtained from the direction information processing unit 106. It outputs to 107. The drawing display processing unit 107 to which the panorama development image and the azimuth information are input draws so that the direction of the display screen can be understood, and outputs to the display device 110 for display.

  FIG. 6 shows a screen display example of the panorama development image and orientation information at this time. When the user changes the opening position of the all-round annular image by the operation unit 108, the panorama development image and the display of the azimuth information can be changed in real time.

  Further, as shown in FIG. 7, when the user changes the position indicating the orientation information of the panorama development image displayed on the display device 110 by the operation unit 108, the orientation information is displayed at the display position of the panorama development image. Can be changed in real time. This is a method of displaying an image when the panorama development conversion process is performed, but instead of displaying all of the panorama development image, a part of the panorama development image may be cut out and displayed. In this case, the display method using the above-described plane perspective projection conversion image is used.

  Also in the case of this panorama development conversion process, when recording as a still image, the azimuth information is recorded in the image recording unit 109 in the GPS Img Direction item of the GPS Info Tag portion of the EXIF attached information together with the panorama development image. As a result, when the panorama development image recorded later is displayed, this data can be used to display it together with the direction information.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (computer program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various computer-readable storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program.

100 omnidirectional camera, 101 omnidirectional mirror unit, 102 imaging sensor unit, 103 azimuth sensor unit, 104 development processing unit, 105 image geometric transformation process, 106 azimuth information processing unit, 107 drawing display processing unit, 108 operation unit, 109 image Recording unit, 110 display, 120 control unit

Claims (12)

Comprising an omnidirectional imaging means having means for imaging an omnidirectional image, an azimuth obtaining means for obtaining azimuth information, and a control means for controlling the operation of the entire apparatus,
The control means acquires azimuth information about a certain reference direction of the screen of the omnidirectional image captured by the omnidirectional imaging means by the azimuth acquisition means and attaches it to the omnidirectional image as metadata. An omnidirectional imaging device characterized by the above. Comprising a perspective projection conversion means for generating a perspective projection conversion image by performing perspective projection conversion on a region of the omnidirectional image captured by the omnidirectional imaging means;
The control means calculates the orientation information of the subject of the perspective projection conversion image generated by the perspective projection conversion means from the orientation information of the reference direction of the omnidirectional image acquired by the orientation acquisition means, The omnidirectional imaging apparatus according to claim 1, wherein the calculated azimuth information is attached to the perspective projection conversion image.   The control means acquires, by the azimuth acquisition means, a display of a perspective projection converted image obtained by perspective projection conversion of a region of an omnidirectional image photographed by the omnidirectional imaging means in real time by the perspective projection conversion means. The omnidirectional imaging apparatus according to claim 2, wherein the azimuth information of the subject of the perspective projection conversion image calculated in real time from the azimuth information of the reference direction of the omnidirectional image is displayed. . Panorama development conversion means for generating a panoramic development image by panoramic development conversion of the omnidirectional image photographed by the omnidirectional imaging means;
The control means uses the azimuth information of the reference direction of the omnidirectional image acquired by the azimuth acquisition means, and the azimuth information of the portion of the panoramic development image generated by the panoramic development conversion means that serves as the reference of the subject The omnidirectional imaging apparatus according to any one of claims 1 to 3, wherein the calculated azimuth information is attached to the panoramic development image.   The control means is acquired by the azimuth acquisition means for the display of a panoramic development image obtained by panoramic development conversion by the panoramic development conversion means in real time in a certain area of the omnidirectional image photographed by the omnidirectional imaging means. 5. The omnidirectional information of the panorama developed image calculated in real time from the azimuth information of the reference direction of the omnidirectional image is displayed. Azimuth imaging device. Comprising an omnidirectional imaging step having a step of capturing an image of omnidirectional, an azimuth acquisition step of acquiring azimuth information, and a control step of controlling the operation of the entire apparatus;
The control step acquires azimuth information about a certain reference direction of the screen of the omnidirectional image captured in the omnidirectional imaging step in the azimuth acquisition step, and attaches it to the omnidirectional image as metadata. An omnidirectional imaging method. A perspective projection conversion step of generating a perspective projection conversion image by performing perspective projection conversion on a region of the omnidirectional image captured in the omnidirectional imaging step;
The control step calculates the orientation information of the subject of the perspective projection conversion image generated in the perspective projection conversion step from the orientation information of the reference direction of the omnidirectional image acquired in the orientation acquisition step, The omnidirectional imaging method according to claim 6, wherein the calculated azimuth information is attached as metadata to the perspective projection conversion image.   In the azimuth acquisition step, the control step acquires a region of the omnidirectional image captured in the omnidirectional imaging step in real time with respect to the display of the perspective projection conversion image obtained by perspective projection conversion in the perspective projection conversion step. The omnidirectional imaging method according to claim 7, wherein the azimuth information of the subject of the perspective projection conversion image calculated in real time from the azimuth information of the reference direction of the omnidirectional image is displayed. . A panoramic development conversion step of generating a panoramic development image by converting panoramic development of the omnidirectional image taken in the omnidirectional imaging step;
In the control step, the azimuth information of the portion of the panorama development image generated in the panorama development conversion step is used as the reference information of the subject in the panorama development conversion step based on the azimuth information of the omnidirectional image obtained in the azimuth acquisition step. The omnidirectional imaging method according to any one of claims 6 to 8, wherein the omnidirectional imaging method is calculated and the calculated azimuth information is attached to the panoramic development image.   The control step is acquired in the azimuth acquisition step with respect to the display of the panorama development image obtained by panorama development conversion in the panorama development conversion step in real time for a certain area of the omnidirectional image photographed in the omnidirectional imaging step. The omnidirectional information according to claim 9, wherein the azimuth information of the portion serving as a reference of the subject of the panoramic developed image calculated in real time from the azimuth information of the direction serving as the reference of the omnidirectional image is displayed. Imaging method. A program for causing a computer to execute an omnidirectional imaging process having a process of capturing an image of an omnidirectional, an azimuth acquisition process for acquiring azimuth information, and a control process for controlling the operation of the entire apparatus,
In the control step, azimuth information about a certain reference direction on the screen of the omnidirectional image captured in the omnidirectional imaging step is acquired in the azimuth acquisition step and attached to the omnidirectional image as metadata. A program characterized by controlling a computer to do so.   A computer-readable storage medium storing the program according to claim 11.

Images