WO2012096168A1 - Image generation device, image generation method, and computer program - Google Patents

Abstract

Provided is an image generating device with which it is possible to reduce computation load and memory access when generating a flat image, cutting an image of a cut range from a full area image while moving the cut range. An image generating device (1) which generates a flat image in a cut range which is cut from a full area image having distortion comprises: a full area image acquisition unit (11) which acquires a full area image; a cut range designation unit (14) which designates a cut range within the full area image; a reuse region calculation unit (17) which derives a reuse region wherein past flat images from the cut range can be reused and a new region wherein past images cannot be reused; a flat image accumulation unit (20) for accumulating the past flat images; a distortion compensation unit (18) which carries out a distortion compensation for the new region of the full area image; and a compositing unit (19) which generates the flat image by compositing the image of the reuse region in the flat images which are accumulated in the flat image accumulation unit (20) and the image of the new region which is distortion compensated with the distortion compensation unit, and outputs the generated flat image to the flat image accumulation unit (20).

Description

Image generating apparatus, image generating method, and computer program Related applications

This application claims the benefit of Patent Application No. 2011-004894 filed in Japan on January 13, 2011, and the contents of the application are incorporated herein by reference.

The present invention relates to an image generation apparatus and an image generation method for generating a planar image for a cutout range cut out from a distorted omnidirectional image, and more particularly to an efficient cutout range when the cutout range is moved within the omnidirectional image. The present invention relates to an image generation apparatus and an image generation method for generating a flat image.

Conventionally, all-around cameras are used for surveillance cameras. The omnidirectional camera is also referred to as a 360-degree camera, and uses a fisheye lens or a mirror to shoot an omnidirectional image with a horizontal angle of view of 360 degrees and a vertical angle of view of 180 degrees to obtain a circular or donut-shaped image (hereinafter, referred to as “round-angled camera”) "All-around image") can be obtained.

補正 By correcting the entire surrounding image to a flat image, it is possible to obtain an image similar to that taken with a normal camera. In particular, if the all-around image is developed for the entire circumference, a 360-degree panoramic image can be obtained by one shooting with one camera.

In addition, a partial range of the entire surrounding image is cut out and corrected to a flat image, and by moving the cut out range, a plurality of flat images (or the same as those shot while panning and tilting a normal camera) Moving image) is obtained. In this way, the control for changing the cutout range of the all-around image so as to obtain a plurality of planar images obtained by shooting a normal camera while panning, tilting, and zooming is performed using the electronic PTZ ( pan, tilt, zoom) control.

Examples of uses for extracting a part of the entire surrounding image and moving the extraction range include, for example, a surveillance camera that manually moves the extraction range, a movement that automatically moves the extraction range in a predetermined trajectory, There are those that detect a moving object from the entire surrounding image and automatically move the extraction range so as to track the detected moving object.

FIG. 10A is a diagram illustrating an example of an all-around image, and FIG. 10B is a diagram illustrating a planar image generated by cutting out a partial range PA from the all-around image in FIG. 10A. is there. As shown in FIG. 10A, the omnidirectional image is captured using a fish-eye lens, for example, by an equidistant projection method in which the distance from the center of the omnidirectional image and the elevation angle are proportional to each other. Yes. Therefore, in order to cut out a partial range PA from the entire surrounding image and obtain the planar image shown in FIG. 10B, it is necessary to perform distortion correction (see, for example, Patent Document 1).

11 and 12 are diagrams for explaining generation of a planar image. FIG. 11A shows a virtual hemispherical surface (hereinafter simply referred to as “hemispherical surface”) SS having the center of the fisheye lens as its center and the focal length as its radius, and a plane for projecting the entire peripheral image. It is a figure which shows the relationship with a certain virtual camera plane VP, FIG.11 (b) is a figure which shows the omnidirectional image projected on image sensor IS. 12 (a) and 12 (b) are planes passing through the center O of FIG. 11 (a) and perpendicular to the virtual camera plane VP. It is the figure which cut | disconnected virtual camera plane VP (Hereafter, FIG. 2, 5, 6, 7 is also the same). FIG. 12A shows the virtual camera plane VP, and FIG. 12B shows the virtual camera plane VP ′ when the cutout range is moved.

JP 2008-61260 A

Processing to generate a planar image by cutting out a partial range from the entire surrounding image and performing distortion correction (that is, projecting onto a virtual camera plane) has a problem that it requires a large amount of computation and memory access, and the processing load is large. . If the cutout range is fixed at one place, the amount of computation and memory access can be reduced by reusing the pixels of the previous frame for the next frame after generating a flat image of a frame. .

However, when moving the extraction range, the pixels in the previous frame cannot be reused as they are. As is clear from FIG. 12 (b), the cutout range is moved so that the virtual camera plane is shown from the virtual camera plane VP in FIG. 12 (a) (shown by a dotted line in FIG. 12 (b)). When moving to the virtual camera plane VP ′ of 12 (b) (shown by a solid line in FIG. 12 (b)), reusable pixels are limited to pixels on the intersection line of both planes, and the amount of computation And memory access cannot be reduced sufficiently.

The present invention has been made in view of the above-described problems, and reduces the amount of computation and memory access when generating a planar image by extracting an image of the extraction range from the entire surrounding image while moving the extraction range. An object of the present invention is to provide an image generating apparatus capable of performing the above.

An image generation apparatus according to the present invention is an image generation apparatus that generates a planar image for a cutout range cut out from a distorted omnidirectional image, and specifies an omnidirectional image acquisition unit that acquires the omnidirectional image, and a cutout range. A cutout range specification unit, a reuse region that can reuse past flat images from the cutout range, a reuse region calculation unit that calculates new regions that cannot reuse past flat images, and to store past flat images A flat image storage unit, a distortion correction unit that performs distortion correction on a new area of the entire surrounding image, an image of a reuse area in the flat image stored in the flat image storage unit, and a distortion correction unit It has a configuration including a combining unit that generates a planar image by combining the corrected image of the new region and outputs it to the planar image storage unit.

An image generation method of the present invention is an image generation method for generating a planar image for a cutout range cut out from a distorted omnidirectional image, and specifies an omnidirectional image acquisition step for acquiring the omnidirectional image, and a cutout range. For a cutout area specification step, a reuse area calculation step for obtaining a reuse area in which a past flat image can be reused from the cutout area, a new area in which a past flat image cannot be reused, and a new area of the entire surrounding image A distortion correction step for performing distortion correction, a plane image generation step for generating a plane image, and a plane image storage step for storing the plane image generated in the plane image generation step. The image of the reusable area in the past flat image accumulated in the image accumulation step and the new area corrected in distortion in the distortion correction step By combining the image, and generates a planar image.

The computer program of the present invention is a computer program for causing a computer to execute the above image generation method.

Since the present invention reuses a past plane image for a reuse area in order to generate a plane image, the distortion correction unit only needs to correct the distortion only for the new area. Can be reduced.

As described below, there are other aspects of the present invention. Accordingly, the disclosure of the present invention is intended to provide part of the invention and is not intended to limit the scope of the invention described and claimed herein.

FIG. 1 is a configuration diagram of an image generation apparatus according to an embodiment of the present invention. 2A is a diagram illustrating a conventional method for generating a planar image. FIG. 2B is a diagram illustrating a method for generating a planar image by the image generating apparatus according to the embodiment of the present invention. FIG. 3A is a diagram showing the relationship between the cutout range at time t0 and its virtual camera plane and the cutout range at time t1 in the embodiment of the present invention. FIG. 3B shows the embodiment of the present invention. FIG. 3C is a diagram showing the relationship between the virtual camera plane at time t0 and the cut-out range and virtual camera plane at time t1 in FIG. 3C. The virtual camera plane at time t0 and the virtual camera at time t1 in the embodiment of the present invention The figure which shows the relationship between a plane and an overlap area and a non-overlap area FIG. 4 (a) is a diagram showing a motion area in an omnidirectional image in the embodiment of the present invention. FIG. 4 (b) is a diagram showing a reuse area and a new area. FIG. 5 is a diagram for explaining the reuse limit due to the center angle difference of the cutout range in the embodiment of the present invention. FIG. 6 is a diagram for explaining the reuse limit due to the overlap rate of the virtual camera plane in the embodiment of the present invention. FIG. 7A is a diagram showing a state in which the reuse limit in the embodiment of the present invention has been reached. FIG. 7B is a diagram of a virtual newly obtained after reaching the reuse limit in the embodiment of the present invention. Diagram showing camera plane FIG. 8 is a flowchart of an image generation method executed by the image generation apparatus according to the embodiment of the present invention. FIG. 9 (a) is a diagram showing the relationship between the cutout range at time t0 and its virtual camera plane and the cutout range at time t1 and its virtual camera plane in a modification of the embodiment of the present invention. FIG. 9C shows the relationship between the virtual camera plane at time t0 and the virtual camera plane at time t1 in the modification of the embodiment of the present invention. FIG. 9C shows the virtual at the time t0 in the modification of the embodiment of the present invention. The figure which shows the relationship between a camera plane and the virtual camera plane in the time t1, and an overlapping area and a non-overlapping area. FIG. 10A is a diagram showing an example of an all-around image. FIG. 10B is a diagram showing a planar image generated by cutting out a cut-out range from the all-around image in FIG. 10A. FIG. 11A is a diagram showing the relationship between the hemispherical surface and the virtual camera plane. FIG. 11B is a diagram showing an omnidirectional image projected on the image sensor. 12A is a diagram showing the virtual camera plane. FIG. 12B is a diagram showing the virtual camera plane when the cutout range is moved.

The detailed description of the present invention will be described below. The embodiments described below are merely examples of the present invention, and the present invention can be modified in various ways. Accordingly, the specific configurations and functions disclosed below do not limit the scope of the claims.

An image generation apparatus according to an embodiment of the present invention is an image generation apparatus that generates a planar image for a cutout range cut out from an omnidirectional image having distortion, an omnidirectional image acquisition unit that acquires an omnidirectional image, and a cutout A cutout range designating unit for designating a range, a reusable area for reusing a past flat image from the cutout range, a reusable area calculating unit for finding a new area where a past flat image cannot be reused, and a past flat image A flat image storage unit for storing image data, a distortion correction unit for performing distortion correction on a new area of the entire surrounding image, an image of a reuse area in the flat image stored in the flat image storage unit, and distortion correction The image forming apparatus includes a combining unit that generates a plane image by combining the image of the new area whose distortion has been corrected by the unit and outputs the image to the plane image storage unit.

With this configuration, in order to generate a planar image, since the past planar image that has already been generated is reused for the reuse region, the distortion correction unit only needs to correct the distortion only for the new region. The amount of computation and memory access can be reduced.

Further, in the above image generation device, the past plane image may be a plane image of the previous frame.

With this configuration, since the plane image of the previous frame is reused as the past plane image, the reuse area is increased, and the effect of reducing the amount of computation and memory access is increased.

Further, in the above-described image generation apparatus, the reuse area calculation unit may obtain an area within a cutout range where a past plane image is present as a reuse area.

Thus, when a past plane image has already been generated in the cutout range, the past plane image can be reused for the area, and the amount of computation and memory access can be reduced.

In the above image generation device, the distortion correction unit may perform distortion correction by projecting the entire surrounding image of the new area onto the virtual camera plane using the virtual camera plane, and the virtual used by the distortion correction unit. The camera plane may overlap with the virtual camera plane when the past plane image is generated.

With this configuration, since the reuse area and the new area are images projected on the same plane, it is possible to eliminate the unnaturalness caused by the mismatch of the projection planes between the reuse area and the new area.

In addition, the image generation apparatus may further include a motion detection unit that detects a motion region in the entire surrounding image, and the reuse region calculation unit excludes the motion region and includes a past plane image within a cutout range. The area may be a reuse area.

With this configuration, when there is a moving object such as a person who moves in the entire surrounding image, the area is excluded from the reuse area and becomes a new area. A situation where an accurate image cannot be obtained can be avoided.

The image generation apparatus may further include a reusability determination unit that determines reusability according to a predetermined determination criterion, and the combining unit is determined to be reusable by the reusability determination unit. Only in such a case, the image of the reuse area stored in the flat image storage unit and the image of the new area whose distortion has been corrected by the distortion correction unit may be combined to generate a flat image.

This configuration can reduce inaccuracy and excessive image quality degradation of the generated plane image by forcibly reusing the past plane image even when it is not appropriate to reuse the past plane image.

Further, in the above image generation device, when it is determined that the reusability determination unit determines that reuse is not possible, the distortion correction unit performs distortion correction on the entire surrounding image for the entire region of the cutout range, and the combining unit The image that has been subjected to distortion correction by the distortion correction unit may be output as a flat image to the flat image storage unit.

With this configuration, when it is determined that reuse is not possible, a flat image is generated by performing distortion correction on the entire surrounding image as usual without reusing the previous flat image, so the cutout range is moved. Each time the reuse limit is reached, a reliable plane image can be generated, and after such a plane image is generated, past plane images can be reused until the reuse limit is reached again. Can do.

Further, in the above image generation device, the determination criteria are the center direction of the cutout range and the center direction of the current cutout range when the flat image is generated by performing distortion correction on the entire surrounding image for all the regions of the cutout range. The reusability determination unit may determine that reusability is not possible when the angle difference is equal to or greater than a predetermined threshold.

With this configuration, it is possible to avoid an excessive distortion of the planar image generated by reusing the past planar image.

Further, in the above image generation apparatus, the distortion correction unit may perform distortion correction by projecting the entire surrounding image of the new area onto the virtual camera plane using the virtual camera plane. It is possible to overlap the virtual camera plane when the plane image is generated, and the determination criterion is a criterion of the overlapping rate between the virtual camera plane when the past plane image is generated and the virtual camera plane overlapping with it. The reusability determination unit may determine that reusability is not possible when the duplication rate is equal to or less than a predetermined threshold.

With this configuration, when the virtual camera plane overlap rate is too small and the value of reusing past plane images is low, a plane image can be generated using only the entire surrounding image.

Further, in the above image generation apparatus, the determination criterion may be a criterion for the size of the reuse area, and the reusability determination unit determines that the reuse is not possible when the reuse area is equal to or less than a predetermined threshold. You may judge.

With this configuration, when the size of the reusable area is too small and the value of reusing a past planar image is low, a planar image can be generated using only the entire surrounding image.

In the above image generation device, the determination criterion may be a reference for a time difference between the time when the planar image is generated using only the entire surrounding image and the current time. May be determined not to be reusable when is equal to or greater than a predetermined threshold.

This configuration can avoid an excessive difference from the actual all-around image.

An image generation method according to an embodiment of the present invention is an image generation method for generating a planar image with respect to a cutout range cut out from an omnidirectional image having distortion, an omnidirectional image acquisition step for acquiring an omnidirectional image, and a cutout A cutout range designating step for designating a range, a reuse area for reusing a past flat image from the cutout range, a reusable area calculating step for finding a new area for which a past flat image cannot be reused, Including a distortion correction step for performing distortion correction on a new region, a plane image generation step for generating a plane image, and a plane image storage step for storing the plane image generated in the plane image generation step. In the step, the image of the reuse area in the past flat image accumulated in the flat image accumulation step and the distortion correction in the distortion correction step are performed. By combining the image of the new area, to produce a planar image.

Even with this configuration, in order to generate a planar image, the past planar image is reused for the reuse region. Therefore, the distortion correction unit only needs to correct the distortion only for the new region. Access can be reduced.

The computer program according to the embodiment of the present invention is a computer program for causing a computer to execute the above-described image generation method.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

First, the basic principle by which the image generation apparatus according to the embodiment of the present invention solves the conventional problem will be described with reference to FIG. FIG. 2A shows a conventional plane image generation method, and FIG. 2B shows a plane image generation method by the image generation apparatus of the present embodiment.

As already described as the background art, in the conventional planar image generation method, as shown in FIG. 2 (a), in a certain frame, a planar image is generated from the entire surrounding image in the cutout area CA1, and in the next frame. When the cut-out range changes from the cut-out range CA1 to the cut-out range CA2, even if an attempt is made to reuse the pixels of the flat image in the cut-out range CA1 for the flat image in the cut-out range CA2, the virtual camera plane VP1 and the virtual camera plane VP2 Only pixels on the intersection line can be reused. This is because the normal direction of the virtual camera plane VP1 and the virtual camera plane VP2 is always the direction from the center of the cutout range, that is, from the center of the cutout range to the center of the hemisphere.

On the other hand, in the present embodiment, as shown in FIG. 2B, a virtual camera plane VP2 ′ for generating a planar image of the cutout area CA2 is within the cutout area CA2 and pixels are arranged. The virtual camera plane VP2 ′ and the virtual camera plane VP1 are partially overlapped (on the same plane) in parallel with the virtual camera plane VP1 of the plane pixel to be reused. That is, the virtual camera plane is obtained so as to be parallel to and partially overlap the virtual camera plane of the previous frame.

Then, when generating a plane image in which the entire surrounding image of the cutout range CA2 is projected on the virtual camera plane VP2 ′, the virtual camera plane VP1 is obtained for the overlapping portion of the virtual camera plane VP2 ′ and the virtual camera plane VP1. It is possible to reuse the pixels of the planar image of the cutout range CA1 that has been projected and already generated.

In this way, in the present embodiment, when the cutout range moves, the virtual camera plane is set so that it is within the moved cutout range and partially overlaps the virtual camera plane of the reference image. Therefore, a partial area of the reference image can be reused when generating a planar image of the cutout range after movement, and the amount of computation and memory access can be reduced. Note that the image of the portion that does not overlap the virtual camera plane of the previous frame is generated by projecting the entire surrounding image onto the virtual camera plane.

Even when the cutout range further moves, the virtual camera plane that overlaps with the virtual camera plane of the previous frame is obtained until the reuse limit described later, and the overlapped portion is already generated by the previous one. Reuse the plane image of the frame, and for the part that does not overlap with the virtual camera plane of the previous frame, project the corresponding part of the omnidirectional image on the virtual camera plane of the current frame, and Is generated.

The above is the basic principle by which the image generation apparatus of the present embodiment solves the problem. Hereinafter, the configuration of the image generation apparatus according to the present embodiment will be specifically described.

FIG. 1 is a block diagram showing a configuration of the image generation apparatus according to the present embodiment. The image generation apparatus 1 includes an omnidirectional image acquisition unit 11, an omnidirectional image storage unit 12, a motion detection unit 13, a cutout range designation unit 14, a virtual camera plane calculation unit 15, an overlapping region calculation unit 16, and a reuse region calculation unit 17. A distortion correction unit 18, a synthesis unit 19, and an image storage unit 20.

The all-around image acquisition unit 11 captures the subject and acquires the all-around image. In the present embodiment, a subject image is projected onto an image sensor using a fisheye lens by an equidistant projection method, and an all-around image having a horizontal field angle of 360 degrees and a vertical field angle of 180 degrees is acquired. This all-around image is a circular image, and the degree of distortion increases as the distance from the center increases. The all-around image acquired by the all-around image acquisition unit 11 is output to the all-around image storage unit 12 and the motion detection unit 13.

The cutout range designation unit 14 designates a cutout range (cutout range) for generating a planar image from the entire surrounding image. The cutout range is specified by four vectors respectively directed to four points on the hemisphere, and the range surrounded by the four vectors is the cutout range.

In this embodiment, since the aspect ratio and orientation of the cutout range are fixed, the cutout range is defined by a vector from the center of the hemisphere to the center of the cutout range and the size of the cutout range. The horizontal angle of this vector corresponds to the pan angle of a normal camera, the vertical angle of the vector corresponds to the tilt angle of a normal camera, and the size of the clipping range (cutting angle of view) is usually This corresponds to the zoom of the camera. Therefore, the designation of the extraction range is, in other words, designation of pan, tilt, and zoom in the electronic PTZ control.

The cutout range designation unit 14 may designate a cutout range in accordance with a user operation. In this case, the image generation apparatus 1 is provided with an operation unit, and the user can specify the extraction range by operating the operation unit. The cutout range designation unit 14 may automatically move the cutout range so that the cutout range draws a predetermined trajectory.

Further, the cutout range specifying unit 14 may detect a moving object from the entire surrounding image and specify the cutout range so as to track the moving object. In this case, the cutout range designation unit 14 is notified of the range in which the motion has been detected from the motion detection unit 13, and the cutout range designation unit 14 designates the cutout range so that the range in which the movement has occurred is the center. . Further, the size (zoom magnification) of the cutout range specified by the cutout range specifying unit 14 may be fixed or variable. Further, the cutout range specifying unit 14 specifies a rectangle other than a rectangle or an arbitrary shape on the hemisphere by specifying a plurality of vectors directed to a plurality of points on the hemisphere when specifying the cutout range. Can also be specified.

The plane image storage unit 20 stores information about the plane image and the virtual camera plane for the previous frame. The plane image of the previous frame is output from the synthesis unit 19 to the plane image storage unit 20. Further, the virtual camera plane information of the previous plane image is output from the virtual camera plane calculation unit 15 to the plane image storage unit 20.

Further, after the reuse area calculation unit 17 described later determines that the plane image of the previous frame cannot be reused, the plane image storage unit 20 does not reuse the plane image of the previous frame. The center of the cutout range (hereinafter referred to as “initial cutout range”) when a plane image is generated using only the surrounding image and the shooting time thereof are stored. The center of the initial cutout range and the photographing time are output from the cutout range designation unit 14 to the planar image storage unit 20.

The virtual camera plane calculation unit 15 is based on the virtual camera plane information of the plane image of the previous frame stored in the plane image storage unit 20 and the extraction range information specified by the extraction range specification unit 14. Calculate the virtual camera plane of the frame.

Specifically, the virtual camera plane calculation unit 15 is in the cutout range specified by the cutout range specifying unit 14 and has the same inclination as the virtual camera plane of the plane image of the previous frame, A plane that partially overlaps the virtual camera plane of the plane image of the previous frame is calculated as the virtual camera plane of the current frame.

The virtual camera plane calculated in this way corresponds to the virtual camera plane VP ′ in the example of FIG. The virtual camera plane calculation unit 15 outputs the calculated virtual camera plane information to the overlap region calculation unit 16, the reuse region calculation unit 17, the distortion correction unit 18, and the plane image storage unit 20.

The virtual camera plane is defined by adding the normal direction information to the cut-out range information. In other words, the parameters that define the virtual camera plane are the vector from the center of the hemisphere to the center of the cutout range, which is the parameter that defines the cutout range, and the size of the cutout range plus the normal direction of the virtual camera plane. It is. Note that a parameter of the size (number of pixels) of an image finally obtained as a planar image can be further added. In the present embodiment, it is assumed that the size of the planar image is fixed.

The overlapping area calculation unit 16 reads out information on the virtual camera plane of the previous plane image stored in the plane image storage unit 20 and based on this and the virtual camera plane input from the virtual camera plane calculation unit 15. Thus, a region where both planes overlap each other (hereinafter referred to as “overlapping region”) and a region where both planes do not overlap (hereinafter referred to as “non-overlapping region”) are obtained.

FIG. 3 is a diagram for explaining an example of a method for obtaining an overlapping area in the overlapping area calculation unit 16. 3A to 3C, the times t0 and t1 have a relationship of t0 <t1. Time t1 is present and time t0 is past. Time t1 is a time at which a planar image is generated as the next frame of the planar image generated at time t0.

FIG. 3A is a diagram showing the relationship between the cutout range CAt0 and its virtual camera plane VPt0 at time t0 and the cutout range CAt1 at time t1. FIG. 3B is a diagram showing the relationship between the virtual camera plane VPt0 at time t0, the cutout range CAt1 and the virtual camera plane VPt1 'at time t1. FIG. 3C is a diagram showing the relationship between the virtual camera plane VPt0 at time t0 and the virtual camera plane VPt1 'at time t1, and the overlapping area and the non-overlapping area.

As shown in FIG. 3A, when the cutout range CAt0 at time t0 changes to the cutout range CAt1 at time t1, the overlapping area calculation unit 16 includes the cutout range CAt1 in the cutout range CAt1, as shown in FIG. A virtual camera plane VPt1 ′ at time t1 is generated so as to be parallel to and overlap with the virtual camera plane VPt0 at time t0.

Then, as shown in FIG. 3C, the area overlapping the virtual camera plane VPt1 ′ in the virtual camera plane VPt0 and the area overlapping the virtual camera plane VPt0 in the virtual camera plane VPt1 ′ are determined as overlapping areas. To do. Further, in the cutout range at time t1, a portion corresponding to a region not overlapping with the virtual camera plane VPt0 at time t0 in the virtual camera plane VPt1 'at time t1 is determined as a non-overlapping region. The overlapping area calculation unit 16 outputs the overlapping area and the non-overlapping area obtained as described above to the reuse area calculation unit 17.

The motion detection unit 13 takes a difference between the all-around image at time t0 and the all-around image at time t1, and detects a region where the difference is equal to or greater than a predetermined threshold as a motion region. The omnidirectional image at time t 0 is stored in the omnidirectional image storage unit 12 after being acquired by the omnidirectional image acquisition unit 11. The motion detection unit 13 reads the past all-around image from the all-around image storage unit 12. The all-around image at time t1 is input from the all-around image acquisition unit 11 to the motion detection unit 13.

FIG. 4A is a diagram showing a motion region in the entire surrounding image. In FIG. 4A, the all-around image EPt0 at time t0 and the all-around image EPt1 at time t1 are displayed in an overlapping manner. As shown in FIG. 4A, when a moving person moves between time t0 and time t1, an area where a person is shown at time t0 and an area where a person is shown at time t1 are This is an area where the difference is greater than or equal to the threshold value between the all-around image EPt0 at time t0 and the all-around image EPt1 at time t1. The motion detection unit 13 detects this region as the motion region MA.

The reuse area calculation unit 17 cannot reuse the area in which the plane image of the previous frame can be reused (hereinafter referred to as “reuse area”) and the plane image of the previous frame. An area (hereinafter referred to as a “new area”) in which only the all-around image obtained by the all-around image obtaining unit 11 must be used is obtained. The reuse area calculation unit 17 sets an area that is a combination of a non-overlapping area and a motion area as a new area, and sets an area that is an overlap area and is not a motion area as a reuse area.

FIG. 4B is a diagram showing a reuse area and a new area. As shown in FIG. 4B, the reuse region calculation unit 17 uses the overlap region and the non-overlap region acquired from the overlap region calculation unit 16 and the motion region acquired from the motion detection unit 13 from the overlap region. The area obtained by subtracting the motion area (the area hatched in FIG. 4B) is set as the reuse area. Further, the reuse area calculation unit 17 sets an area obtained by subtracting the reuse area from the virtual camera plane VPt1 'at time t1 as a new area.

The reuse area calculation unit 17 further determines whether or not reuse is possible. That is, the reuse area calculation unit 17 has a function as a reuse permission determination unit. When the predetermined condition exceeds the reuse limit (reuse limit), the reuse area calculation unit 17 does not reuse the plane image (whole) of the previous frame (cannot be reused). The determination is made and a notice to that effect is sent to the distortion correction unit 18. If the predetermined condition does not exceed the reuse limit, the reuse area calculation unit 17 notifies the distortion correction unit 18 of the reuse area and the new area. The reuse area calculation unit 17 determines whether or not reuse is possible according to the following criteria.

(1) Criteria regarding center angle difference of cutout range FIG. 5 is a diagram for explaining a reuse limit due to the center angle difference of the cutout range. As described above, the virtual camera plane calculation unit 15 is part of the virtual camera plane VPt0 at time t0 that is in the cutout range and generates the plane image of the previous frame at time t1 (current). An overlapping virtual camera plane VPt1 ′ is obtained.

At this time, when the angle of the center CCn of the current cutout range CAn with respect to the center CC0 of the intended cutout range CA0 is equal to or greater than a predetermined threshold, the reuse area calculation unit 17 obtains a planar image of the previous frame. It is determined not to be reused (cannot be reused).

If the angle difference between the initial cutout range and the current cutout range becomes too large, the angle between the current normal virtual camera plane VPn and the modified virtual camera plane VPn ′ will become too large accordingly. This is because the distortion of the planar image generated by being projected onto the virtual camera plane VPn ′ becomes too large. The reuse area calculation unit 17 acquires information about the center CC0 of the initial cutout range CA0 from the planar image storage unit 20.

(2) Criteria for Virtual Camera Plane Overlap Rate FIG. 6 is a diagram for explaining the reuse limit based on the virtual camera plane overlap rate. The reuse area calculation unit 17 determines the virtual camera plane VPt1 of the overlapping area between the virtual camera plane VPt1 ′ at the time t1 (current) obtained by the overlapping area calculation unit 16 and the virtual camera plane VPt0 at the time t0 of the previous frame. A ratio (overlap rate) to 'is obtained, and when the overlap rate is equal to or less than a predetermined threshold, it is determined that the plane image of the previous frame is not reused (reusability is not possible).

This is because when the overlap rate is small, it is not necessary or necessary to reuse the plane image of the previous frame until the virtual camera plane is corrected to be parallel to the previous frame. .

(3) Criteria regarding the size of the reuse area As described above, the reuse area calculation unit 17 obtains the reuse area by subtracting the motion area from the overlap area. As a result of subtracting the motion area from the overlapping area, the reuse area calculation unit 17 does not reuse the plane image of the previous frame (unusable for reuse) when the remaining reuse area is smaller than a predetermined threshold. judge.

If the reusable area is small, it is not necessary or valuable to reuse the plane image of the previous frame until the virtual camera plane is corrected to be parallel to the previous frame. is there. The reuse area calculation unit 17 determines the size of the reuse area based on the number of pixels included in the reuse area.

(4) Time Criteria As described above, when generating a planar image, the planar image generated in the previous frame is reused for the reuse area. However, even if the area of the reuse area required in relation to the previous frame is sufficiently large, it is highly worth reusing the planar image of the previous frame, and the planar image is obtained using only the entire surrounding image. If the time interval from the time of generating is increased, there is a possibility that the error in the planar image will increase in the accumulation.

Accordingly, the reuse area calculation unit 17 determines that the plane image of the previous frame when the difference between the time when the plane image is generated using only the entire surrounding image and the current time is equal to or greater than a predetermined threshold. Is not reused (cannot be reused). The reuse area calculation unit 17 acquires the time when the plane image is generated using only the entire surrounding image from the plane image storage unit 20.

In the reuse area, the entire periphery image is used as a new area in the image portion generated when the plane image is generated using only the entire periphery image and each frame up to the previous frame thereafter. Each generated image portion is included. That is, an image to be reused has an old part and a relatively new part. Therefore, by using a criterion regarding time, the old part (which may be a pixel unit or a block unit) may be gradually used as a new area so that it is not reused.

If any of the above criteria (1) to (4) is satisfied, the reuse area calculation unit 17 assumes that the reuse limit has been reached and does not reuse the plane image of the previous frame ( It is determined that it cannot be reused. When the reuse limit is reached, there is no need to adopt a virtual camera plane parallel to the virtual camera plane of the plane image of the previous frame in order to generate a new plane image. 17 obtains the virtual camera plane again. That is, the reuse area calculation unit 17 also has a function as a virtual camera plane recalculation unit that recalculates the virtual camera plane.

When recalculating the virtual camera plane, the reuse area calculation unit 17 can set a plane that is within the cutout range and perpendicular to the center direction as the virtual camera plane, such as the virtual camera plane VP1 of FIG. . Further, the reuse area calculation unit 17 predicts a future cutout range based on the change history of the cutout range, and generates a new reference image after reaching the reuse limit based on the prediction. A virtual camera plane can be determined. Hereinafter, a method for obtaining a virtual camera plane for a new reference image based on prediction will be described.

FIG. 7 is a diagram for explaining a method of obtaining a virtual camera plane based on prediction when the reuse limit is reached. FIG. 7A is a diagram illustrating a state in which the reuse limit is reached, and FIG. 7B is a diagram illustrating a virtual camera plane that is newly obtained after reaching the reuse limit. When the reuse area calculation unit 17 that functions as a virtual camera plane recalculation unit reaches the reuse limit, it predicts the moving direction of the future cutout range from the past cutout history.

In the present embodiment, since the center CC0 of the initial cutout range is stored in the planar image accumulation unit 20 as information on the past cutout range, the reuse area calculation unit 17 performs the center CC0 of the initial cutout range. When the reuse limit is exceeded from (current), the cutout range is predicted to move in the direction toward the center CCn of the cutout range.

The reuse area calculation unit 17 generates a new reference image having a slope symmetrical to the virtual camera plane of the initial cutout range and a plane within the current cutout range with respect to the center CCn of the current cutout range. For the virtual camera plane. That is, as shown in FIG. 7 (b), the reuse area calculation unit 17 sets the reuse limit just between the center CC0 of the initial cutout range and the perpendicular PL drawn from the center of the hemisphere to the new virtual camera plane VPn. The inclination of the new virtual camera plane VPn is determined so that the center CCn of the current (current) cutout range is located.

The distortion correction unit 18 is notified of the reuse region and the new region from the reuse region calculation unit 17 and recycles a part of the plane image of the previous frame. An omnidirectional image is acquired from the acquisition unit 11 and projected onto the virtual camera plane generated by the virtual camera plane unit 15 to generate a new area portion of the plane image, which is output to the synthesis unit 19. .

When receiving the notification that the reference image is not reused and the input of the recalculated virtual camera plane from the reuse area calculation unit 17, the distortion correction unit 18 obtains the all-around image for the entire cutout range. And project it onto the recalculated virtual camera plane to generate a planar image. The generated planar image is output to the planar image storage unit 20 as a new reference image and is also output to the synthesis unit 19.

When only the new area portion of the planar image is received from the distortion correction unit 18, the synthesizing unit 19 reads the planar image of the previous frame from the planar image storage unit 20, and converts the new area of the read planar image to the distortion correction unit. By replacing the image with the image input from 18, both images are combined to generate a planar image. The generated planar image is output to the outside of the image generating apparatus 1 and is stored in the planar image storage unit 20.

When the entire planar image is input from the distortion correcting unit 18 (when the distortion correcting unit 18 generates a planar image using only the entire surrounding image for the entire cutout range), the combining unit 19 The data is output to the outside of the image generation apparatus 1 and the planar image storage unit 20 as it is.

FIG. 8 is a flowchart of an image generation method executed by the image generation apparatus 1. Hereinafter, the image generation method of the present embodiment will be described with reference to FIG. In the image generation method according to the present embodiment, first, an omnidirectional image is captured by the omnidirectional image acquisition unit 11 (step S11). Then, the cutout range designation unit 14 designates the cutout range (step S12).

At this time, the extraction range specifying unit 14 may specify the extraction range based on the all-around image (detecting movement), or may specify the extraction range regardless of the all-around image. In the latter case, the order of step S11 and step S12 may be either first or simultaneous.

Next, the virtual camera plane calculation unit 15 sets the extraction range specified by the extraction range specification unit 14 based on the extraction range specified by the extraction range specification unit 14 and the virtual camera plane of the previous frame. A virtual camera plane that is parallel to the virtual camera plane of the previous frame and partially overlaps is calculated (step S13). The virtual camera plane unit 15 reads the virtual camera plane of the previous frame from the plane image storage unit 20 and writes the generated virtual camera plane to the plane image storage unit 20.

Next, the overlapping area calculation unit 16 overlaps and non-overlapping areas between the virtual camera plane calculated by the virtual camera plane calculation unit 15 and the virtual camera plane of the previous frame read from the plane image storage unit 20. Is calculated (step S14). Here, it is determined whether or not there is an overlapping area (step S15). If there is an overlapping area (YES in step S15), the motion detection unit 13 performs motion detection (step S16). If there is no overlapping area as a result of calculating the overlapping area by the overlapping area calculation unit 16 (NO in step S15), the plane image of the previous frame cannot be reused, and the process proceeds to step S23. To do.

After the motion detection, the reuse region calculation unit 17 obtains a reuse region by subtracting the motion region from the overlap region based on the information on the overlap region and the non-overlap region and the motion detection result, and the motion in the overlap region A region obtained by combining the region and the non-overlapping region is obtained as a new region (step S17). Subsequently, the reuse area calculation unit 17 determines whether or not reuse is possible (step S18). For this determination, the determination criteria (1) to (4) described above are employed. As a result of the determination, when it is determined that any of (1) to (4) and the reuse limit has been reached (NO in step S18), the process proceeds to step S23.

Note that the determination criterion (1) can be determined at the time when the cut-out range is specified. Therefore, if the determination criterion (1) is determined after step S12 and the reuse limit has been reached, that point The process may immediately proceed to step S23. In addition, since the criterion (2) can be determined at the time when the virtual camera plane is calculated, if the criterion (2) is determined after step S13 and the reuse limit is reached, You may transfer to step S23 immediately at the time.

Furthermore, since the determination criterion (4) can be determined at the time of step S11, if the determination criterion (4) is determined after step S11 and the reuse limit has been reached, the step is immediately performed at that time. You may transfer to S23. Regarding the criterion (3), since the information of the reuse area is used, it can be determined only after step S17. In the present embodiment, determination is made for all the determination criteria (1) to (4) after step S17 in which determination can be made for all the determination criteria (1) to (4).

When it is determined that the plane image of the previous frame can be reused (YES in step S18), the distortion correction unit 18 reads out the entire surrounding image from the all-around image acquisition unit 11 for the new region. Then, the distortion is corrected by projecting it onto the virtual camera plane calculated in step S13 (step S19).

Then, the synthesizing unit 19 reads the plane image of the previous frame from the plane image storage unit 20 (step S20), and the partial image obtained in step S19 for the new area in the plane image read in step S20. The two images are combined by replacing (step S21). The synthesizer 20 outputs a planar image (step S22).

If there is no overlapping area as a result of the calculation of the overlapping area (NO in step S15), or if it is determined that reuse is not possible as a result of the reusability determination (NO in step S18), the virtual camera plane is reused. Calculate (step S23).

When the virtual camera plane is recalculated, the distortion correction unit 18 corrects the distortion of the omnidirectional image by projecting the omnidirectional image on the recalculated plane for all regions, and generates a flat image. (Step S24). In this case, the center of the cutout range (initial cutout range) and the photographing time at that time are stored in the planar image storage unit 20 (step S25). Note that either step S24 or step S25 may be performed first or simultaneously. The planar image generated in this way is output in step S22.

When the generated planar image is output in step S22, the process returns to step S11 to capture the entire surrounding image again, and the above processing is repeated.

As described above, according to the image generation device 1 and the image generation method of the present embodiment, when the cutout range moves, the virtual camera that is parallel to the virtual camera plane of the previous frame and partially overlaps Since the planar image projected on the plane is generated, the planar image generated in the previous frame can be reused for the overlapping region. As a result, the area for distortion correction can be reduced, and the amount of computation and memory access can be reduced accordingly.

Further, in the above embodiment, the motion detection unit 13 detects a motion area in the entire surrounding image, and the plane image of the previous frame is reproduced only for the area excluding this motion area from the overlapping area. Since it is used, if it is premised that movement occurs in the entire surrounding image in the usage mode, it is possible to obtain a correct planar image of the portion where the movement has occurred.

In the usage mode, if it is not premised that movement occurs in the all-around image (including the case where it is not necessary to accurately display the generated movement in a sequentially generated planar image), The image storage unit 12 and the motion detection unit 13 can be omitted, and accordingly, the reuse area calculation unit 17 may use the overlap area as a reuse area as it is.

In the above-described embodiment, the reuse area calculation unit 17 has a function as a reuse permission determination unit. When the reuse limit is reached, the reuse of the plane image of the previous frame is reused. Therefore, it is possible to avoid excessive image deterioration and image inaccuracy due to reuse of the plane image of the previous frame.

The image generation device 1 is formed by providing a part of the configuration to the imaging device (camera), the other part of the configuration to the image processing device, and connecting the imaging device and the image processing device. Good. In this case, for example, the imaging apparatus includes the omnidirectional image acquisition unit 11 and the transmission / reception unit, and the image processing apparatus includes the other components and the transmission / reception unit described above.

The transmission / reception unit of the image processing apparatus transmits information on the new area calculated by the reuse area calculation unit 17 to the imaging apparatus. The transmission / reception unit of the imaging apparatus receives the information on the new area and transmits the entire surrounding image of the new area to the image processing apparatus. The transmission / reception unit of the image processing apparatus receives the all-around image of the new area.

In the embodiment described above, a virtual camera plane that is parallel to the virtual camera plane of the previous frame and partially overlaps is calculated, and a new area is acquired by the all-around image acquisition unit 11. Although the calculated omnidirectional image is projected onto the calculated virtual camera plane, the virtual camera plane for projecting the omnidirectional image for the new area may always be a plane perpendicular to the center of the cutout range.

In this case, in the planar image generated by synthesis, the boundary between the new area and the reuse area becomes unnatural, but the inclination of the virtual camera plane increases as the cutout range moves as in the above embodiment. The plane image of the previous frame can be reused without becoming large and without worrying about the reuse limit of the above embodiment.

In the above embodiment, the plane image of the previous frame is reused. However, the plane image generated using only the entire surrounding image is used as a reference image, and the subsequent frames are limited to the reuse limit. Until that time, the reference image may be reused for the reuse area.

In the above embodiment, the method of determining the overlapping area by the overlapping area calculation unit 16 using FIG. 3 has been described, but the method of determining the overlapping area is not limited to this. FIG. 9 is a diagram for explaining another example of a method for obtaining an overlapping area in the overlapping area calculation unit 16.

FIG. 9A shows the relationship between the cutout area CAt0 and its virtual camera plane VPt0 at time t0 and the cutout area CAt1 and its virtual camera plane VPt1 at time t1. FIG. 9B is a diagram illustrating the relationship between the virtual camera plane VPt0 at time t0 and the virtual camera plane VPt1 at time t1. FIG. 9C is a diagram illustrating the relationship between the virtual camera plane VPt0 at time t0 and the virtual camera plane VPt1 at time t1, and the overlapping area and the non-overlapping area.

When the cutout range CAt0 at time t0 changes to the cutout range CAt1 at time t1 as shown in FIG. 9A, the overlapping area calculation unit 16 causes the virtual camera plane VPt1 at time t1 as shown in FIG. 9A. Ask for. The virtual camera plane VPt1 is a plane that is in the cutout range CAt1 and whose normal line overlaps with a vector from the center O of the hemisphere toward the center of the cutout range CAt1 at time t1. That is, in the example of FIG. 3, the virtual camera plane VPt1 ′ that is parallel to and overlaps the virtual camera plane VPt0 at the time t0 is obtained at time t1, but in the example of FIG. 9, the cutout range CAt1 is given. Sometimes, the virtual camera plane VPt1 having no inclination with respect to the cutout range CAt1 is generated regardless of the orientation of the virtual camera plane VPt0 at time t0.

Next, the overlapping area calculation unit 16 extends from the virtual camera plane VPt0 at time t0 to the end point of the virtual camera plane VPt1 at time t1 that overlaps the virtual camera plane VPt0 (shown as a point in FIG. 3 but actually a side). On the other hand, a vertical line (shown as a line in FIG. 3 but actually a plane) is dropped, and the virtual camera plane at time t1 from the end point of the virtual camera plane VPt0 at time t0 that overlaps the virtual camera plane VPt1 at time t1. A perpendicular line is drawn to VPt1.

Then, as shown in FIG. 9C, the area overlapping the virtual camera plane VPt1 in the virtual camera plane VPt0 and the area overlapping the virtual camera plane VPt0 in the virtual camera plane VPt1 are determined as overlapping areas. Further, in the cut-out range at time t1, a portion corresponding to a region that does not overlap with the virtual camera plane VPt0 at time t0 in the virtual camera plane VPt1 at time t1 is determined as a non-overlapping region.

Note that the image generation apparatus and the image generation method of the above-described embodiment may be realized by a general-purpose computer executing a computer program.

Although the presently preferred embodiments of the present invention have been described above, various modifications can be made to the present embodiments, and such modifications are within the true spirit and scope of the present invention. It is intended that the appended claims include all modifications.

As described above, the present invention can reduce the amount of computation and memory access, cuts out a partial range of a captured image having distortion, corrects the distortion, and generates a planar image of the partial range. This is useful as a generation method.

DESCRIPTION OF SYMBOLS 1 Image generation apparatus 11 All-around image acquisition part 12 All-around image storage part 13 Motion detection part 14 Cutout range designation | designated part 15 Virtual camera plane calculation part 16 Overlapping area calculation part 17 Reuse area calculation part 18 Distortion correction part 19 Synthesis | combination part 20 Flat image storage unit

Claims (13)

1. An image generation device that generates a planar image for a cut-out range cut out from an all-around image having distortion,
   An omnidirectional image acquisition unit for acquiring the omnidirectional image;
   A cutout range designating unit for designating the cutout range;
   A reuse area calculation unit for obtaining a reuse area in which a past plane image can be reused from the cutout range, and a new area in which a past plane image cannot be reused;
   A plane image storage unit for storing the past plane images;
   A distortion correction unit that performs distortion correction on the new area of the entire surrounding image;
   By combining the image of the reuse area in the plane image accumulated in the plane image accumulation unit and the image of the new area corrected for distortion by the distortion correction unit, a plane image is generated, A combining unit for outputting to the planar image storage unit;
   An image generation apparatus comprising:
2. 2. The image generation apparatus according to claim 1, wherein the past plane image is a plane image of a previous frame.
3. 3. The image generation apparatus according to claim 1, wherein the reuse area calculation unit obtains an area within the cutout range where the past plane image is present as the reuse area.
4. The distortion correction unit performs distortion correction by projecting the entire surrounding image of the new area on the virtual camera plane using a virtual camera plane,
   4. The image generation apparatus according to claim 1, wherein a virtual camera plane used by the distortion correction unit overlaps a virtual camera plane when the past plane image is generated. 5.
5. A motion detection unit for detecting a motion region in the entire surrounding image;
   5. The reuse area calculation unit according to claim 1, wherein an area within the cutout range where the past plane image is present is defined as the reuse area except for the motion area. 6. The image generating apparatus described in 1.
6. A reusability determination unit that determines reusability according to a predetermined determination criterion;
   The composition unit corrects distortion in the image of the reuse area stored in the planar image storage unit and the distortion correction unit only when the reuse determination unit determines that the reuse is possible. 4. The image generation apparatus according to claim 1, wherein a planar image is generated by combining the image of the new area. 5.
7. If it is determined that the reuse is not possible in the reusability determination unit, the distortion correction unit performs distortion correction on the entire surrounding image for the entire region of the cutout range, and the synthesis unit performs the distortion correction. The image generation apparatus according to claim 6, wherein an image on which distortion correction has been performed by the unit is output to the planar image storage unit as the planar image.
8. The determination criterion is that the angular difference between the center direction of the cut-out range and the center direction of the current cut-out range when the flat image is generated by performing distortion correction of the all-around image for the entire region of the cut-out range. Is the standard of
   The image generation apparatus according to claim 6, wherein the reusability determination unit determines that reusability is not possible when the angle difference is equal to or greater than a predetermined threshold.
9. The distortion correction unit performs distortion correction by projecting the entire surrounding image of the new area on the virtual camera plane using a virtual camera plane,
   The virtual camera plane overlaps with the virtual camera plane when generating the past plane image,
   The determination criterion is a criterion of an overlapping rate between the virtual camera plane when the past plane image is generated and the virtual camera plane overlapping therewith,
   The image generation apparatus according to claim 6, wherein the reusability determination unit determines that reusability is not possible when the duplication rate is equal to or less than a predetermined threshold.
10. The criterion is a criterion for the size of the reuse area,
    The image generation apparatus according to claim 6, wherein the reusability determination unit determines that reusability is not possible when the reuse area is equal to or less than a predetermined threshold.
11. The determination criterion is a reference for the time difference between the time when the planar image is generated using only the entire surrounding image and the current time,
    The image generation apparatus according to claim 6, wherein the reusability determination unit determines that reusability is not possible when the time difference is equal to or greater than a predetermined threshold.
12. An image generation method for generating a planar image for a cut-out range cut out from an all-around image having distortion,
    An all-around image obtaining step for obtaining the all-around image;
    A cutting range specifying step for specifying the cutting range;
    A reuse area calculation step for obtaining a reuse area in which a past plane image can be reused from the cutout range, and a new area in which a past plane image cannot be reused;
    A distortion correction step for correcting distortion for the new region of the all-around image;
    A plane image generation step for generating a plane image;
    A plane image accumulation step for accumulating the plane image generated in the plane image generation step;
    Including
    The plane image generation step combines the image of the reuse area in the past plane image accumulated in the plane image accumulation step and the image of the new area whose distortion is corrected in the distortion correction step. An image generation method characterized by generating the planar image.
13. A computer program for causing a computer to execute the image generation method according to claim 12.
    

Images