JP2023042257A - Image processing device, image processing method and program - Google Patents

Abstract

To prevent cut-off of a cut-out image while suppressing the restriction of a cut-out range.SOLUTION: An image processing device according to one aspect comprises: first acquisition means which acquires a reference image; second acquisition means which acquires a peripheral image obtained by imaging a peripheral area on the periphery of an imaging area of the reference image; conversion means which converts the peripheral image so as to extend a field angle in the time of imaging of the reference image; and composition means which generates a composite image obtained by combining a converted image converted from the peripheral image and the reference image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing device, an image processing method, and a program.

In a sports scene, many videos are acquired by many cameras, edited and distributed. It is also important to analyze the contents of the play from the acquired video, to know the characteristics of the opponent, and to improve the strength of the own team.
However, since these processes require a lot of manpower, images may not be used effectively when targeting students and youth. For this reason, there is a demand for a technology that acquires video without manpower to create video for distribution and extracts information necessary for sports analysis.

In fast-moving scenes such as sports, it is difficult to control the camera according to the movement of the subject. Know how to create.
At that time, a large amount of memory is required in order to acquire high-resolution moving images over a wide range, and the processing load becomes heavy. Moreover, it is wasteful to acquire even a motionless area as a moving image.

Patent Literature 1 discloses a video display device that, when displaying a program with a moving image and a still image divided into a plurality of systems, cuts out portions of the same subject from the moving image and inserts them into the still image. there is

Patent Document 2 describes an image for adjusting the alignment of a moving image with respect to a panoramic image based on control points for associating the moving image with the area of the panoramic image by extracting feature points between the panoramic image and the moving image. A display device is disclosed.

Japanese Patent Application Laid-Open No. 2002-200001 discloses an imaging device that displays a finder image and also displays a frame that is an index indicating a moving image imaging area in the finder image.

JP-A-2005-229451 JP 2014-82764 A Japanese Patent Application Laid-Open No. 2003-259161

However, with the methods disclosed in Patent Literatures 1 and 2, there is a possibility that parting may occur depending on the clipping range. In addition, Patent Document 3 does not disclose image clipping.
The problem to be solved by the present invention is to prevent cutout images from being cut off while suppressing restrictions on the cutout range.

An image processing apparatus according to one aspect of the present invention includes first acquisition means for acquiring a reference image, second acquisition means for acquiring a peripheral image in which a peripheral area of an imaging area of the reference image is captured, and the reference image. transforming means for transforming the peripheral image so as to expand the angle of view at the time of imaging; .

According to one aspect of the present invention, it is possible to prevent cutout images from being cut off while suppressing restrictions on the cutout range.

1 is a block diagram showing a configuration example of an image processing system according to a first embodiment; FIG. 2 is a block diagram showing a functional configuration example of the image processing apparatus according to the first embodiment; FIG. 2 is a block diagram showing a hardware configuration example of an image processing apparatus according to the first embodiment; FIG. FIG. 4 is a diagram showing an example of a clipping area set in a reference image; FIG. 3B is a view showing an example of a clipped image clipped from the clipping region of FIG. 3A; 4 is a flowchart showing image processing of the image processing apparatus according to the first embodiment; FIG. 4 is a diagram showing an example of a peripheral area around a reference image; FIG. 10 is a diagram showing an example of a synthesized image obtained by synthesizing a reference image and a converted image; FIG. 10 is a diagram showing an example of a clipping area set in a composite image; FIG. 6B is a view showing an example of a clipped image clipped from the clipping region of FIG. 6A; 4A and 4B are diagrams for explaining image conversion processing of the image processing apparatus according to the first embodiment; FIG. FIG. 2 is a block diagram showing a functional configuration example of an image processing apparatus according to a second embodiment; FIG. 10 is a block diagram showing an example hardware configuration of an image processing apparatus according to the second embodiment; 8 is a flowchart showing image processing of the image processing apparatus according to the second embodiment; FIG. 11 is a diagram showing an example of setting boundaries of clipping regions according to the second embodiment; FIG. 11 is a diagram showing an example of presentation of the position of a reference image during zooming according to the second embodiment; FIG. 11 is a diagram showing a setting example of a moving object region according to the second embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the embodiments are essential for solving means of the present invention. The configuration of the embodiment can be appropriately modified or changed according to the specifications of the device to which the present invention is applied and various conditions (use conditions, use environment, etc.). The technical scope of the present invention is determined by the claims and is not limited by the following individual embodiments.

In each of the following embodiments, the imaging of sports will be described as an example, but not limited to this, it can be applied to imaging of various events, concerts, lecture scenes, attractions, parades, acrobats, dances, plays, meteor showers, and the like. is also possible. Further, in each of the following embodiments, an image processing apparatus functioning as an image capturing apparatus (network camera) capable of connecting to a network and communicating with other apparatuses will be taken as an example. However, it is not limited to this, and can also be applied to an image processing apparatus that functions as an imaging apparatus that cannot be connected to a network. Further, in each of the following embodiments, the image processing apparatus will be described as having an imaging function. , and the image processing device may acquire a captured image from another device.

<First embodiment>
In the present embodiment, by obtaining a still image around an imaging area captured as a moving image, correcting the still image according to the imaging angle of view of the moving image, and holding the still image, the area of interest can be extracted from the moving image at high speed. A method of complementing processing will be described. By performing such processing, when cutting out a moving image from a bird's-eye view image, it is possible to create a seamless image even in an area with large distortion in the peripheral portion.

FIG. 1 is a block diagram showing a configuration example of an image processing system according to the first embodiment.
In FIG. 1, an image processing system 10 includes an image processing device 100 functioning as an imaging device and a client device 200. The image processing device 100 and the client device 200 are connected via a network 300 so as to be able to communicate with each other. be.

In this embodiment, a device (a network camera, etc.) that allows the image processing device 100 to connect to the network 300 and communicate with other devices is taken as an example. However, it is not essential that the image processing apparatus 100 can be connected to the network 300 . For example, the image processing apparatus 100 and the client apparatus 200 may be directly connected by a cable. For such cables, for example, HDMI (registered trademark) (High-Definition Multimedia Interface) and SDI (Serial Digital Interface) can be used.

The client device 200 transmits a delivery request command requesting delivery of a video (image) stream and a setting command for setting various parameters to the image processing device 100 based on a user's operation. The client device 200 can be realized by installing a predetermined program in a computer such as a personal computer, tablet terminal, or smart phone.

The image processing apparatus 100 distributes the video stream to the client apparatus 200 according to the distribution request command, and stores various parameters according to the setting command. Further, the image processing apparatus 100 acquires a reference image and a peripheral image, converts the peripheral image so as to expand the angle of view when the reference image is captured, and converts the converted image converted from the peripheral image and the reference image. Generate a synthesized composite image. At this time, the image processing apparatus 100 converts the peripheral image so that the subject protruding from the imaging area into the peripheral area can be continuously connected. The peripheral image is an image obtained by capturing a peripheral area around the imaging area of the reference image. The reference image may be a moving image, and the peripheral images may be still images. This reference image may be a bird's-eye view image captured by a wide-angle camera.

Here, in order to focus on the edge of the bird's-eye view image, if the edge of the bird's-eye image is cut out from the bird's-eye view image, the entire cropped area cannot fit within the angle of view of the wide-angle camera, and the cropped image is cut off. occurs. On the other hand, if clipping is performed from the bird's-eye view image so as not to cause the clipping, the range of the clipped image is limited.
For this reason, when the image processing apparatus 100 cuts out around the edge portion of the bird's-eye view image, the image processing apparatus 100 cuts out from a composite image in which the converted image converted from the surrounding image and the reference image are combined. As a result, the image processing apparatus 100 can prevent the clipped image from being cut off while suppressing restrictions on the clipping range that is clipped as the region of interest.

At this time, when the imaging direction of the reference image and the imaging direction of the peripheral image are different, the image processing apparatus 100 can convert the peripheral image so that the imaging direction of the peripheral image is equal to the imaging direction of the reference image. . As a result, the image processing apparatus 100 can generate a composite image such that the subject protruding from the imaging area into the peripheral area is continuously connected while suppressing the distortion of the subject protruding from the imaging area into the peripheral area. can be done.

For example, the image processing apparatus 100 obtains a still image in which the peripheral area around the imaging area of the moving image is captured by PTZ (pan, tilt, zoom) without moving the installation position of the same camera. It is converted into a planar image of the imaging position and held. When using a method called trapezoidal transformation, in which four vertices are specified and converted into a rectangle, the clipped image is converted from a still image in which the surrounding area is captured to a planar image at the shooting position of the moving image, so that it can be clipped without any breaks. Images can be acquired easily.

2A is a block diagram illustrating a functional configuration example of the image processing apparatus according to the first embodiment; FIG.
Of the functional blocks shown in FIG. 2A, for functions realized by software, a program for providing the function of each functional block is stored in a memory such as a ROM (Read Only Memory). Then, the program is read into a RAM (Random Access Memory) and executed by a CPU (Central Processing Unit). For the functions realized by hardware, for example, by using a predetermined compiler, a dedicated circuit may be automatically generated on the FPGA from a program for realizing the function of each functional block. FPGA is an abbreviation for Field Programmable Gate Array. Also, a gate array circuit may be formed in the same manner as the FPGA and implemented as hardware. Also, it may be realized by an ASIC (Application Specific Integrated Circuit). Note that the configuration of the functional blocks shown in FIG. 2A is an example, and a plurality of functional blocks may constitute one functional block, or one of the functional blocks may be divided into blocks that perform a plurality of functions. good too.

In FIG. 2A , the image processing apparatus 100 includes a peripheral image acquisition section 211 , an image conversion section 212 , a reference image acquisition section 213 , an image analysis section 214 , an image synthesis section 215 and an image clipping section 216 .

A reference image acquisition unit 213 acquires a reference image. The reference image may be a moving image. This reference image may be a bird's-eye view image captured by a wide-angle camera. For example, the reference image acquisition unit 213 may acquire a moving image from the imaging unit 221 in FIG. 2B or an external device (not shown), and use one frame included in the moving image as the reference image. Also, the reference image acquisition unit 213 generates a reference image using various parameters (various settings) acquired from the storage unit 222 in FIG. 2B.

The peripheral image acquisition unit 211 acquires peripheral images. The peripheral image is an image obtained by capturing a peripheral area around the imaging area of the reference image. For example, the peripheral image acquisition unit 211 acquires a still image from the imaging unit 221 in FIG. 2B or an external device (not shown), and generates a peripheral image corresponding to each frame. In addition, the surrounding image acquisition unit 211 generates a surrounding image using various parameters (various settings) acquired from the storage unit 222 in FIG. 2B.

The image conversion unit 212 converts the peripheral image based on the image generated by the peripheral image acquisition unit 211 and various parameters (camera direction, zoom value, etc.) used when acquiring the image. At this time, the image conversion unit 212 converts the surrounding images so that the angle of view at the time of capturing the reference image is expanded. For example, the image conversion unit 212 can convert the peripheral image so that the subject protruding from the imaging area of the reference image into the peripheral area can be continuously connected. Here, when the imaging direction of the reference image and the imaging direction of the peripheral image are different, the image conversion unit 212 can convert the peripheral image so that the imaging direction of the peripheral image becomes equal to the imaging direction of the reference image. . For example, the image conversion unit 212 converts the images into images captured by cameras placed at the same position with different pan/tilt/zoom values. The image conversion unit 212 stores the converted peripheral image generated by the conversion in the storage unit 222 .

The image analysis unit 214 analyzes the reference image and determines the clipping position of the clipped image that is clipped from the composite image. For example, the image analysis unit 214 performs object detection processing on the reference image acquired by the reference image acquisition unit 213, and extracts an image clipping region. The object detection method may be, for example, a method based on machine learning. At this time, the image analysis unit 214 can generate a classifier that has learned the features of the object to be detected, apply the classifier to the image data, detect the object, and determine the cutout position. The image analysis unit 214 retains in the storage unit 222 the reference image acquired from the reference image acquisition unit 213 and the information regarding the clipping region extracted from the reference image.

The image synthesizing unit 215 synthesizes the converted peripheral image held in the storage unit 222 and the reference image obtained by the reference image obtaining unit 213 to generate a synthesized image. The generated composite image is sent to the image clipping unit 216 .

The image clipping unit 216 creates a pseudo PTZ image (hereinafter referred to as a pseudo PTZ image) from the synthesized image generated by the image synthesizing unit 215 based on the information about the clipping area held in the storage unit 222 .

FIG. 2B is a block diagram illustrating a hardware configuration example of the image processing apparatus according to the first embodiment;
In FIG. 2B, the image processing apparatus 100 includes an imaging unit 221, a storage unit 222, a control unit 223, a communication unit 224, and an accelerator unit 225.

The imaging unit 221 acquires a moving image by converting the light imaged on the light receiving surface of the imaging element through the lens into an electric charge for each pixel. The imaging unit 221 includes, for example, a zoom lens, a focus lens, a blur correction lens, an aperture, a shutter, an optical low-pass filter, an IR (Infrared Rays) cut filter, a color filter, an imaging element, and the like. The imaging device may be, for example, an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device).

The storage unit 222 stores programs for the image processing apparatus 100 to perform various operations. The storage unit 222 can also store data (commands and image data) and various parameters acquired from an external device such as the client device 200 via the communication unit 224 . For example, the storage unit 222 stores parameters related to camera orientation and magnification such as pan/tilt/zoom for moving images acquired by the imaging unit 221, and parameters related to camera settings such as camera white balance and exposure. memorize The storage unit 222 may also store parameters related to image data, including the frame rate of image data and the size (resolution) of image data. Further, the storage unit 222 can provide a work area used when the control unit 223 executes various processes. Furthermore, the storage unit 222 can also function as a frame memory or a buffer memory.

Storage unit 222 may be configured with both ROM and RAM, or may be configured with either ROM or RAM. The storage unit 222 uses a storage medium such as an SSD (Solid State Drive), flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, non-volatile memory card, or DVD. good too.

The control unit 223 controls the entire image processing apparatus 100 by executing programs stored in the storage unit 222 . The control unit 223 may control the entire image processing apparatus 100 in cooperation with the programs stored in the storage unit 222 and an OS (Operating System).

The control unit 223 may be a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit). Further, the control unit 223 may include a processor such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit). .

The communication unit 224 transmits and receives wired or wireless signals to communicate with the client device 200 via the network 300 of FIG.

The accelerator unit 225 is a processing unit added to the camera mainly to perform high-performance processing based on machine learning such as Deep Learning. For example, the accelerator section 225 may take charge of the processing of the image analysis section 214 . The accelerator unit 225 can include a CPU, GPU, FPGA, storage unit, and the like.

Note that the present embodiment shows a case where the image processing apparatus 100 including the accelerator unit 225 performs image analysis processing. In addition to this, the analysis processing may be performed by an accelerator unit added by USB (Universal Serial Bus) or the like from the outside. Alternatively, video may be input directly from a camera to a dedicated device having a GPU or FPGA via HDMI (High-Definition Multimedia Interface) or SDI (Serial Digital Interface) and analyzed. Also, once the video is distributed, it may be stored in a PC (Personal Computer) or the like and processed, or the video recorded in the SD card attached to the camera may be processed in a PC that is not connected.

Note that the functional configuration of the image processing apparatus 100 in FIG. 2A may be realized by the hardware shown in FIG. 2B, or may be realized by software.

The operation of the image processing apparatus 100 shown in FIG. 2A will be described below by taking as an example a case of acquiring an overall bird's-eye view image of a sporting event and creating a composite image from which a moving scene can be cut out without interruption. .

FIG. 3A is a diagram showing an example of a bird's-eye view image used as a reference image.
In FIG. 3A, the reference image frame 30 includes a scene in which a plurality of players 310 playing basketball and a basketball court 320 are captured from above. Here, it is assumed that the position of the area to be cut out when creating the cutout image is set as the cutout area 330 by paying attention to the area where the players 310 gather.

FIG. 3B is a diagram showing an example of a clipped image clipped from the clipping region in FIG. 3A.
In FIG. 3B, it is assumed that a clipped image frame 31 is generated from a clipped region 330 set in the reference image frame 30 of FIG. 3A. The clipped image frame 31 can be generated by converting a clipped area 330 surrounded by four vertices specified on the reference image frame 30 into a rectangle. At this time, since pixel data cannot be obtained for a portion where the cutout region 330 protrudes from the reference image frame 30 , the cutout image frame 31 has closed regions 340 and 341 with a pixel value of 0. FIG.

The image processing apparatus 100 acquires a peripheral image in which the peripheral region of the reference image frame 30 is captured, and images the peripheral image so as to be continuously connected to the reference image frame 30 in order to prevent the occurrence of the cut-off regions 340 and 341 . Holds the transformed transformed surrounding image. Then, the image processing apparatus 100 creates a synthesized image by complementing the reference image frame 30 with the converted peripheral images, and cuts out from this synthesized image. As a result, even when the clipped region 330 protrudes from the reference image frame 30, it is possible to prevent the clipped regions 340 and 341 from occurring in the clipped image.

FIG. 4 is a flow chart showing image processing of the image processing apparatus according to the first embodiment. FIG. 5A is a diagram showing an example of a peripheral area around a reference image, and FIG. 5B is a diagram showing an example of a synthesized image obtained by synthesizing the reference image and the converted image. FIG. 6A is a diagram showing an example of a cutout region set in a composite image, and FIG. 6B is a diagram showing an example of a cutout image cut out from the cutout region of FIG. 6A.

Note that each step in FIG. 4 is implemented by the control unit 223 reading and executing the program stored in the storage unit 222 in FIG. 2B. Also, at least part of the flowchart shown in FIG. 4 may be realized by hardware. When implemented by hardware, for example, by using a predetermined compiler, a dedicated circuit may be automatically generated on an FPGA from a program for implementing each step. Also, a Gate Array circuit may be formed in the same manner as the FPGA and implemented as hardware. Also, it may be realized by an ASIC.
In this case, each block in the flowchart shown in FIG. 4 can be regarded as a hardware block. A plurality of blocks may be collectively configured as one hardware block, or one block may be configured as a plurality of hardware blocks.

The processing in FIG. 4 is realized by executing the control program stored in the storage unit 222 by the control unit 223 in FIG. 2B to perform calculation and processing of information and control of each hardware. In addition, the accelerator unit 225 also performs processing in order to perform processing by machine learning at high speed.

In step S41 of FIG. 4, the peripheral image acquisition unit 211 and the reference image acquisition unit 213 acquire settings necessary for generating image data. For example, the peripheral image acquisition unit 211 and the reference image acquisition unit 213 acquire parameters related to the imaging unit 221 that captures images from the storage unit 222 . Parameters related to the imaging unit 221 include the installation position, orientation, angle of view, focal length of the lens, frame rate, and image data size (resolution) of the camera. For example, the image data size can be 1920×1080 pixels and the frame rate can be 30 fps.

Next, in step S42, the peripheral image acquiring unit 211 instructs imaging of the peripheral area with respect to the reference position indicated by the various settings acquired in step S41, and acquires image data corresponding to the peripheral area. The peripheral areas are, for example, areas 50A to 50D around the imaging area of the reference image frame 30 in FIG. 3A, as shown in FIG. 5A. The peripheral area is an area used for interpolation of the image captured at the reference position, and is centered around an area that does not contain important moving object elements. For this reason, for example, when a basketball game is being played, as shown in FIG. 3A, the movement of the player and the ball is imaged at the reference position, whereas the image is captured before the start of the game when there are no players on the court. The surrounding area may be imaged from time to time.

However, the timing of imaging the surrounding area is not limited to before the start of the game, and can be arbitrarily selected. For example, an image may be captured during a break between matches and updated to the latest image. In particular, only when there is a change in lighting or the like over time, the image may be captured and updated during the rest period. Also, the timing of capturing the image of the peripheral area may be after changing the parameters of the imaging unit 221 that captures the peripheral image. In addition, when the camera is installed at a fixed position, the peripheral image acquisition unit 211 holds peripheral images captured under a plurality of conditions according to camera parameters such as exposure. You may select the surrounding image imaged by .

Here, there are multiple methods for acquiring image data corresponding to the peripheral area. For example, the camera or pan head may be rotated to change the orientation of the camera left and right or up and down to capture an image, or the zoom may be changed to capture a wider range than the reference position. The peripheral image acquisition unit 211 generates one or more image frames corresponding to the peripheral area captured in this way, and stores them in the storage unit 222 .

Next, in step S43, the image conversion unit 212 generates a converted peripheral image that can be continuously connected to the reference image frame from the image frames of the peripheral region acquired in step S42. Here, it is assumed that the first imaging direction of the reference image and the second imaging direction of the peripheral images are different. At this time, the image conversion unit 212 maps the plane coordinates of the peripheral image captured from the second imaging direction onto the spherical coordinates, and captures the plane coordinates of the peripheral image mapped onto the spherical coordinates from the first imaging direction. may be mapped onto the plane coordinates of the reference image.

Next, in step S44, the reference image obtaining unit 213 instructs the imaging of the reference image based on the various settings obtained in step S41, and generates the reference image frame 30. FIG.

Next, in step S45, the image analysis unit 214 performs object detection processing on the reference image frame acquired in step S44 to detect a target object. For example, the image analyzer 214 may set the detection targets to be the player 310 and the ball in the reference image frame 30 of FIG. 3A.

As a method of detecting an object by image analysis, a method based on machine learning, particularly Deep Learning, can be used in order to achieve highly accurate real-time processing. Specifically, YOLO (You Only Look Once), SSD (Single Shot Multibox Detector), and the like are examples of object detection methods based on image analysis. SSD is one of methods for detecting each object from an image in which a plurality of objects are shown.

In order to construct an identifier that detects the player 310 and the ball using an SSD, images showing human bodies are collected from a plurality of videos and prepared as learning data. Specifically, the regions of the human body and the ball in the video are extracted, and a file is created in which the coordinates of the center position of the region and the size of the region are described. The training data prepared in this way is learned, and a discriminator for detecting a human body and a ball is constructed.

When the human body and the ball are detected using the generated classifier, information indicating the position of the detected area and its size are obtained. The information indicating the position of the area indicates the center position of the detection area and its rectangular size (width and height) as a ratio to the size of the image, using coordinates with the upper left corner of the frame as the origin. The positions and sizes of the detection areas thus acquired are acquired as a list, since multiple persons may be detected within the frame.

Next, the image analysis unit 214 obtains a cutout area 330 based on the acquired detection results of the player 310 and the ball. Specifically, the center of gravity of the detected positions of the player 310 and the ball is obtained, and the center of the cutout area 330 is set as the center of gravity. Here, there are a plurality of methods for obtaining the center of the cutout region. For example, the image analysis unit 214 may calculate the center of gravity of the player 310 and the ball with the same weight, or may calculate the center of gravity of the player 310 only.

Next, the image analysis unit 214 calculates the 4 vertices of the cutout region 330 . The cropping area 330 is an imaging area when a second camera (a camera with a small angle of view placed at the same position as the bird's-eye camera) is directed to the center of the cropping area 330 obtained above. At this time, a pseudo PTZ image is generated by clipping and performing PTZ with the second camera.

Next, in step S46, the image synthesizing unit 215 synthesizes the converted peripheral image generated in step S43 and the reference image frame obtained in step S44 to generate a synthetic image 52 with a pseudo super-wide angle. do. The transformed peripheral image generated in step S43 is an image that has already been transformed so as to be continuously connectable with the reference image frame. Therefore, the image synthesizing unit 215 can generate the synthetic image 52 by embedding the reference image acquired in step S44 in the transformed peripheral image. Furthermore, the image synthesizing unit 215 obtains parameters for converting the coordinate values on the reference image into the coordinates of the synthetic image. This parameter is preserved as a translation in the x and z directions, depending on the size of the transformed surrounding image. Then, the image synthesizing unit 215 saves the generated synthetic image 52 in the storage unit 222 .

For example, as shown in FIG. 5B, the image synthesizing unit 215 generates a synthesized image 52 by interpolating around the reference image frame 30 with the transformed surrounding images 51A to 51D acquired in step S43.

Next, in step S47, the image clipping unit 216 performs trapezoidal transformation on the synthesized image 52 created in step S46 from the positions of the four vertices of the clipped region 330 obtained in step S45, and obtains a shape as shown in FIG. 6B. , to create a cropped image frame 33 . Here, the image clipping unit 216 converts the coordinates of the four vertices of the clipping region 330 into coordinates on the composite image 52 by using the translation parameters obtained in step 43 as necessary.

Next, in step S48, the control unit 223 determines whether or not there is image data to be clipped. If there is image data to be clipped (Yes in step S48), the control unit 223 returns the process to step S44 and continues the process for the next image data. If there is no image data to be clipped (No in step S48), the control unit 223 ends the process.

FIG. 7 is a diagram for explaining image conversion processing of the image processing apparatus according to the first embodiment.
In FIG. 7A, in the image conversion process, spherical coordinates are set with the position O of the camera 101 as the origin for the reference image frame 30 captured by the bird's-eye view camera. Assuming that the center of the reference image frame 30 is R, the upper direction from the center R is the z coordinate, and the right direction is the x coordinate, the pixel positions on the reference image frame 30 are on the spherical coordinates seen from the position O of the camera 101 one to one. can be mapped to Coordinates Q (x, y, z) on reference image frame 30 are associated with coordinates P (xp, yp, zp) on spherical surface 60 of radius r with position O of camera 101 as the origin.

Here, the spherical coordinates (r, θ, φ) are defined as shown in FIG. 7(B). At this time, when coordinates Q (x, y, z) on the reference image frame 30 are represented by spherical coordinates, they can be given by the following equation (1).

A coordinate P (xp, yp, zp) on the spherical surface 60 having a radius r with the position O of the camera 101 as the origin can be given by the following equation (2).

Next, in order to image the surrounding area of the reference image frame 30, the orientation of the camera 101 is changed as shown in FIG. 7(C). At this time, the center of the captured peripheral image frame 51 moves to R′, and the point T on the peripheral image frame 51 is associated with the point S on the spherical surface 60 . In this way, by mapping the information of the surrounding image frame 51 obtained by imaging the surrounding area of the reference image frame 30 onto the spherical surface 60, the area obtained by extending the reference image frame 30 at the center R can be obtained by using the equation (1) again. can be associated with

The image conversion unit 212 generates a converted peripheral image that can be continuously connected to the reference image frame 30 by image conversion from the peripheral image frame 51 obtained by imaging the peripheral area of the reference image frame 30 . Furthermore, the image conversion unit 212 calculates parameters for converting coordinate values on the reference image frame 30 . Specifically, it holds the size of the transformed surrounding image (the number of pixels added to the left and top of the reference image), and holds the pixel position as a value to translate when specifying the top left of the image as the reference. . In this manner, the image transforming unit 212 stores the generated transformed peripheral image and the translation parameter in the storage unit 222 .

The image clipping unit 216 generates a pseudo PTZ image by clipping, as shown in FIG. 7(D). Specifically, the image clipping unit 216 converts the central coordinates of the clipped region 330 obtained on the reference image frame 30 to a point U (θc, φc) on the spherical coordinates by the image conversion shown in step S43 in FIG. Convert to Then, the image clipping unit 216 sets the horizontal angle of view of the size to be clipped to 2Δθ and the vertical angle of view to 2Δφ with the point U as the center, and the clipping region 61 on the spherical surface 60 as shown in the following equation (3). 4 vertices (F1, F2, F3, F4).

Then, the image clipping unit 216 converts the four vertices (F1, F2, F3, F4) of the clipped region 61 acquired on the spherical surface 60 to the coordinates on the reference image frame 30 using the equation (1) again, Obtained as the four vertices of the clipped region 330 . The image clipping unit 216 saves the acquired coordinates of the four vertices of the clipping region 330 in the storage unit 222 .

As described above, according to the first embodiment described above, the image processing apparatus 100 generates a clipped image frame (pseudo PTZ image) from an image (reference image frame) to be captured as a moving image. The surrounding area of the frame is imaged in advance. Then, the image processing apparatus 100 holds an image of the peripheral area captured in advance as a transformed peripheral image that has been image-transformed so as to be continuously connected to the reference image frame, and combines it around the reference image frame to create a synthesized image 52 . create. By creating the cropped image frame 33 from the composite image 52, the image processing apparatus 100 can prevent the out-of-sight areas 340 and 341 in FIG. 3B without changing the cropped area 330 in FIG. 3A.

<Second embodiment>
In the first embodiment, when creating a pseudo PTZ image, the transformed peripheral images are image-transformed so as to be continuously connected to the reference image frame 30 so as not to cause the out-of-view areas 340 and 341 in FIG. 3B. I explained how to complement using .
At this time, it is preferable to present to the user in an easy-to-understand manner how much of the surrounding area should be imaged. Also, when the converted peripheral image is obtained as a still image, the complementary region becomes a past still image. A still image of the wall of the gymnasium above the reference image frame 30 in FIG. 3A does not cause a sense of incongruity.

In the second embodiment, after acquiring the transformed peripheral image, the reference image and the transformed peripheral image are displayed, and parameters such as the moving image capturing range are set. Specifically, the image processing apparatus 100 assists the setting of parameters related to the generation of the pseudo PTZ image by indicating changes in the pan/tilt possible range due to the complementary area, changes in the reference image area when zoomed, and the like.
In addition, in 2nd Embodiment, the same code|symbol is attached|subjected about the part similar to 1st Embodiment, and detailed description is abbreviate|omitted.

FIG. 8A is a block diagram showing a functional configuration example of an image processing apparatus according to the second embodiment.
In FIG. 8A, the image processing apparatus 101 includes a setting value acquisition unit 716 instead of the image clipping unit 216 of the image processing apparatus 100 in FIG. 2A. Otherwise, the image processing apparatus 101 can be configured in the same manner as the image processing apparatus 100 .

The setting value acquisition unit 716 acquires settings displayed on the display screen of the composite image. The settings are the boundaries of the cropped image that is cropped from the composite image, or the position of the reference image based on zoom, pan or tilt. The setting may be the range of the moving object area of the synthesized image, or processing for the moving object area. The processing for the moving object region is enhancement processing of the transformed image included in the clipped image clipped from the composite image. It is also possible to set how to combine moving images and still images. For example, the setting value obtaining unit 716 displays the synthesized image created by the image synthesizing unit 215 on the GUI unit 726 in FIG. 8B. Then, the setting value acquisition unit 716 acquires setting parameters necessary for creating a pseudo PTZ image based on user input.

FIG. 8B is a block diagram illustrating a hardware configuration example of an image processing apparatus according to the second embodiment;
In FIG. 8B, the image processing apparatus 101 has a GUI (Graphical User Interface) unit 726 added to the image processing apparatus 100 in FIG. 2B. Otherwise, the image processing apparatus 101 can be configured in the same manner as the image processing apparatus 100 . A GUI unit 726 indicates a GUI for displaying a synthesized image and interactively obtaining input from the user.

FIG. 9 is a flow chart showing image processing of the image processing apparatus according to the second embodiment.
The processing in FIG. 9 is realized by executing the control program stored in the storage unit 222 by the control unit 223 in FIG. 8B to perform calculation and processing of information and control of each hardware. Note that the processing other than steps S85 and S86 is the same as the processing in FIG. 4, so description thereof will be omitted.

In step S85, the image synthesizing unit 215 in FIG. 8A synthesizes the converted peripheral image generated in step S43 and the reference image frame obtained in step S44 to generate a synthetic image 52 with a pseudo super-wide angle. do. Then, the control unit 223 presents the synthesized image created by the image synthesizing unit 215 to the user through the GUI unit 726 .

Next, in step S86, the setting value acquisition unit 716 acquires the user's setting values for the synthesized image, and acquires the setting parameters necessary for creating the pseudo PTZ image.

FIG. 10A is a diagram illustrating a setting example of boundaries of clipping regions according to the second embodiment.
In FIG. 10A, as setting parameters, for example, the left and right boundaries 910 of the range that the cutout region 330 can take can be set on the synthesized image 53 . Also, although not shown in the figure, upper and lower boundaries of the range that the clipping area 330 can take can also be set on the synthesized image 53 . The composite image 53 may be a scene in which the player 310 and the like are not yet on the basketball court 320 before the start of the match. At this time, in the synthesized image 53, the reference image frame 32 of FIG. 5A can be used instead of the reference image frame 30 of the synthesized image 52 of FIG. 5B.

FIG. 10B is a diagram showing a presentation example of the position of the reference image during zooming according to the second embodiment.
In FIG. 10B, as a setting parameter, for example, a position 920 when the angle of view of the reference image is changed by zooming can be shown. As a setting parameter, the position when the reference image is moved by panning or tilting can also be indicated.

FIG. 11 is a diagram showing a setting example of a moving object area according to the second embodiment.
In FIG. 11, a moving body region 321 in which the presence of a moving body is expected is set on the synthesized image 53, and the set value acquisition unit 716 can acquire the setting. The composite image 53 includes the image of the basketball court 320 captured by the bird's-eye camera, but there are regions in the lower left and right sides of the basketball court 320 that are not included in the angle of view. Replacing such an area with a transformed surrounding image generated from a past still image will cause the player to disappear when the player enters the area. In order to prevent such a situation, when the inside of the moving object region 321 is replaced with the converted peripheral image, it is possible to draw the user's attention by highlighting, for example, by coloring. Alternatively, it is possible to track the moving object of the player and change the processing depending on whether or not it is predicted that the player is in an area where the player appears to disappear. Allowing the user to make choices regarding such processing changes.

As described above, according to the above-described second embodiment, the image processing apparatus 100 provides a GUI so that the user can easily specify setting parameters for creating a pseudo PTZ image from a reference image and transformed peripheral images. presented to Thereby, the image processing apparatus 100 can create a pseudo PTZ image in a form desired by the user.

(Other embodiments)
The present invention can also be implemented by processing in which one or more processors read and execute a program that implements one or more functions of the above-described embodiments. The program may be supplied to a system or device having a processor via a network or storage medium. The invention can also be implemented in a circuit (eg, an ASIC) that implements one or more of the functions of the embodiments described above.

10 image processing system, 100 image processing device, 200 client device, 300 network

Claims (13)

a first acquisition means for acquiring a reference image;
a second acquiring means for acquiring a peripheral image in which a peripheral area around the imaging area of the reference image is captured;
conversion means for converting the peripheral image so as to expand the angle of view when the reference image is captured;
synthesizing means for generating a synthetic image by synthesizing the transformed image transformed from the peripheral image and the reference image;
An image processing device comprising: 2. The image processing apparatus according to claim 1, wherein said conversion means converts said peripheral image so that a subject protruding from said imaging area into said peripheral area can be continuously connected. When the first imaging direction of the reference image and the second imaging direction of the peripheral image are different, the conversion means converts the second imaging direction of the peripheral image to be equal to the first imaging direction of the reference image. 3. The image processing apparatus according to claim 2, wherein the peripheral image is transformed. The conversion means maps the plane coordinates of the peripheral image captured from the second imaging direction onto spherical coordinates, and converts the plane coordinates of the peripheral image mapped onto the spherical coordinates from the first imaging direction. 4. The image processing apparatus according to claim 3, wherein mapping is performed on the plane coordinates of the captured reference image. analysis means for analyzing the reference image and determining a cut-out position of the cut-out image cut out from the synthesized image;
clipping means for clipping the clipped image from the composite image based on the clipping position;
5. The image processing apparatus according to any one of claims 1 to 4, further comprising: 6. The peripheral image is an image captured before the start of a sports match, during a sports break, or after a change in a parameter of imaging means for capturing the peripheral image. The image processing device according to . 7. The method according to any one of claims 1 to 6, wherein said second acquiring means acquires a peripheral image captured under an imaging condition closest to an imaging condition of said reference image from peripheral images captured under a plurality of imaging conditions. The image processing device according to any one of items 1 and 2. 8. The image processing apparatus according to any one of claims 1 to 7, further comprising third acquisition means for acquiring settings displayed on the display screen of the composite image. 9. The image processing apparatus according to claim 8, wherein the setting is a boundary of a clipped image clipped from the composite image, or a position of the reference image based on zooming, panning, or tilting. 10. The image processing apparatus according to claim 8, wherein the setting is a range of a moving object area of the synthesized image and a process for the moving object area. 11. The image processing apparatus according to claim 10, wherein the processing for the moving object region is enhancement processing of a transformed image included in a clipped image clipped from the composite image. obtaining a reference image;
obtaining a peripheral image in which a peripheral area around the imaging area of the reference image is captured;
a step of transforming the peripheral image so as to expand the angle of view when the reference image is captured;
generating a synthesized image by synthesizing the transformed image transformed from the surrounding image and the reference image;
An image processing method comprising: A program for operating a computer as the image processing apparatus according to any one of claims 1 to 11.

Images