JP2013106324A - Image processor, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily provide an image equivalent to that photographed at a view point which an appreciator desires.SOLUTION: A picked-up image acquisition section 62 acquires data on a plurality of picked-up images imaged by a plurality of cameras 11 imaging prescribed space where an object OB may move from respective positions. A position information acquisition section 63 acquires a position of the object OB moving in the prescribed space. A view point conversion image processing section 74 processes view point conversion on data of the picked-up image acquired by the picked-up image acquisition section 62 on the basis of the acquired position to generate data of the image in which a virtual view point arranged in the object OB watches a prescribed viewing direction as data of a view point conversion image. A display control section 65 performs control for displaying and outputting data of the generated view point conversion image.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program that make it possible to easily obtain an image equivalent to a photograph taken from a viewpoint desired by a viewer.

  Conventionally, a fixed-point camera, a camera held by a photographer at a news report seat, or the like is used for shooting a video used for broadcasting in a sports competition such as soccer. In other words, sports cameras that are telephoto shot from a distance are broadcast by these cameras.

  However, in such an image captured from a distance, even if telephoto shooting is interwoven, it gives the viewer the impression that it is not powerful.

  Therefore, in order to obtain a more powerful image of a ball game such as soccer, a technique in which a camera or a sensor is incorporated in a ball for ball game has been proposed (for example, see Patent Documents 1 to 3). . By applying such technology, if the sensor in the ball detects an impact or acceleration, the camera in the ball can take an image at a predetermined moment, so that it is possible to reproduce the image from the ball viewpoint. Become. However, since the ball rotates or vibrates, Patent Document 1 discloses a technique for stabilizing an image by controlling the attitude of the falling ball itself.

  In recent years, game machines simulating sports such as soccer and martial arts have also been developed (see, for example, Patent Document 4). In such a game machine, three-dimensional CG (Computer Graphics) simulating a predetermined scene can be reconfigured and replayed.

JP 2007-104254 A JP 2001-042420 A JP 2007-104234 A JP 07-116343 A

  However, in order to obtain the image of the ball viewpoint by applying the conventional techniques of Patent Documents 1 to 3, since the ball is successively changed in posture, the posture control of the ball itself or a built-in camera is sequentially confirmed. Must be done. For this purpose, a large-scale facility with a complicated structure is required, and an image from a ball viewpoint cannot be easily obtained.

  In addition, there are many viewers that request to obtain an image from the viewpoint of not only the ball but also other objects such as players, but the conventional techniques of Patent Documents 1 to 3 meet the request. I can't respond enough.

  In addition, the conventional technique of Patent Document 4 is a technique that can be applied to a virtual reality of a game to the last, and it is difficult to apply the technique as it is to a technique for imaging a real sport or the like.

  The present invention has been made in view of such a situation, and an object thereof is to easily obtain an image equivalent to a photograph taken from a viewpoint desired by a viewer.

  In order to achieve the above object, an image processing apparatus according to an aspect of the present invention provides data of a plurality of captured images captured by each of a plurality of imaging apparatuses that capture a predetermined space in which an object can move from different positions. The plurality of image acquisition means acquired by the image acquisition means based on the position acquired by the image acquisition means, the position acquisition means for acquiring the position of the object moving in the predetermined space, and the position acquisition means Image processing means for generating, as viewpoint conversion image data, image data representing a state in which the virtual viewpoint arranged on the object looks at a predetermined viewing direction by processing viewpoint data of the captured image data of And output control means for executing control for outputting data of the viewpoint conversion image generated by the image processing means.

  According to the present invention, it is possible to easily obtain an image equivalent to that taken from the viewpoint desired by the viewer.

1 is a schematic diagram showing a schematic configuration of an image processing system according to a first embodiment of the present invention. FIG. 2 is a schematic diagram illustrating various specific examples of an original image of a viewpoint conversion image generated by the image processing apparatus of FIG. 1. The example of the viewpoint conversion image produced | generated with the image processing apparatus of FIG. 1 is shown. It is a block diagram which shows the structure of the hardware of the image processing apparatus which concerns on this embodiment. It is a functional block diagram which shows the functional structure for performing a viewpoint conversion image generation process among the functional structures of the image processing apparatus of FIG. 6 is a flowchart illustrating a flow of viewpoint-converted image generation processing according to the first embodiment, which is executed by the image processing apparatus of FIG. 4 having the functional configuration of FIG. 5. It is a schematic diagram which shows schematic structure of the image processing system which concerns on the 1st modification of 1st Embodiment of this invention, Comprising: It is a schematic diagram which shows schematic structure when a player's object is selected as a viewpoint. The example of the viewpoint conversion image produced | generated with the image processing apparatus of FIG. 1 is shown. It is a schematic diagram which shows an example of the acquisition method of the positional information on a selection object. It is a schematic diagram which shows an example of the acquisition method of the positional information on a selection object. It is a schematic diagram which shows an example of the acquisition method of the positional information on a selection object. It is a flowchart explaining the flow of the viewpoint conversion image generation process which concerns on 2nd Embodiment which the image processing apparatus of FIG. 4 which has the functional structure of FIG. 5 performs.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a schematic diagram showing a schematic configuration of an image processing system according to the first embodiment of the present invention.
The image processing system 10 according to the present embodiment is set for the purpose of imaging a soccer game held in the stadium F.
For this reason, m objects that can serve as viewpoints of images provided to the viewer are adopted from soccer balls (hereinafter abbreviated as “balls”), players, and the like. Here, m is an arbitrary integer value of 1 or more, but in this embodiment, it is “7”, which is the same value as the ball and the number of players.

Specifically, FIG. 1 shows a schematic configuration of the image processing system according to the present embodiment when a virtual viewpoint arranged on a ball object is selected.
In this specification, the viewpoint of an imaging unit (such as a camera) that actually captures a real space is referred to as “real viewpoint”, and “virtual viewpoint” is described as a concept for this actual viewpoint. That is, the “virtual viewpoint” refers to a virtual viewpoint that is variably arranged on an object (real object or virtual object) other than the imaging unit.

As shown in FIG. 1, in the image processing system 10 according to the present embodiment, an object OB1 that is a ball, objects OB2 to OB7 that are respective players, and cameras 11a to 11h that shoot the stadium F from the surroundings. The image processing device 12, the relay device 13, and n external terminals 14-1 to 14-n are connected to each other via a network N. The network N is an arbitrary network such as a LAN (Local Area Network) or the Internet.
Hereinafter, when it is not necessary to individually distinguish the external terminals 14-1 to 14-n, these are collectively referred to as “external terminal 14”. Further, when the external terminal 14 is called, the reference numerals of the constituent elements are also omitted in the description.
Further, hereinafter, when it is not necessary to individually distinguish the objects OB1 to OB7, they are collectively referred to as “object OB”. Further, in the case of calling the object OB, the reference numerals of the constituent elements are also omitted in the description.
The object OB includes a sensor unit 111 and a transmission unit 112, which will be described in detail later with reference to FIG. 5, and sequentially transmits position information data of the object OB to the image processing device 12.
In addition, hereinafter, when it is not necessary to individually distinguish the cameras 11a to 11h, these are collectively referred to as “camera 11”. In addition, when the camera 11 is called, the reference numerals of the constituent elements will be described with a to h omitted.

  As shown in FIG. 1, the image processing apparatus 12 can select a ball object OB1 among the objects OB1 to OB7 as an object OB for arranging a virtual viewpoint. The object selected to become the virtual viewpoint in this way is called OB, and hereinafter referred to as “selected object”.

Here, in this embodiment, each of the cameras 11a to 11h is each of the surroundings of the stadium F so that any one point in the stadium F is included in at least one captured image of the cameras 11a to 11h. It is arranged at a position (fixed point). Therefore, in the present embodiment, it is assumed that an image in which the selected object OB is reflected is always captured by at least one of the cameras 11a to 11h.
Therefore, the image processing device 12 processes at least one of the captured image data captured by the cameras 11a to 11h based on position information sequentially transmitted from the selected object OB1 moving in the movement direction A1. As a result, data of a series of images (moving images, continuous shot images, or still images) in which the virtual viewpoint arranged in the selected object OB1 moves in the moving direction A1 without blurring or rotating is generated. To do. That is, in this case, the image processing device 12 generates data of an image (moving image, continuous shot image, or still image) based on the moving direction A1 of the selected object OB1.
Specifically, since the selected object OB1 is a ball, blurring or rotation originally occurs, but the virtual viewpoint can be changed to the selection object OB1 so as to move in the movement direction A1 without blurring or rotation. Be placed. In other words, in the example of FIG. 1, the virtual viewpoint is viewed from the selected object OB1, but is not fixedly placed and does not shake or rotate, so that the horizontal direction can always be viewed in a certain direction ( As a result, the selected object OB1 is variably arranged so as to move in the movement direction A1 without rotating). Here, the predetermined direction viewed from the virtual viewpoint from the selected object OB1 is hereinafter referred to as “view direction”.
On the other hand, the captured image data captured by the cameras 11a to 11h is image data representing a state in which the stadium F including the selected object OB1 is viewed from the periphery from an actual viewpoint.
Therefore, the image processing apparatus 12 has a virtual viewpoint that moves the image data representing the state viewed from the actual viewpoint around (outside) the stadium F in the moving direction A1 of the selected object OB1 in the stadium F. It can also be understood that the moving direction A1 has been converted into image data representing a view as viewed from the viewing direction.
Hereinafter, image data representing a state viewed from an actual viewpoint of the cameras 11a to 11h around the stadium F (external), that is, image data to be converted is referred to as “original image data”. " On the other hand, image data converted from the original image data, that is, an image representing a state where the virtual viewpoint moving in the moving direction A1 of the selected object OB1 in the stadium F is viewed with the moving direction A1 as the viewing direction. The data is referred to as “viewpoint-converted image data”.

  FIG. 2 shows various specific examples of captured images captured by the cameras 11a to 11h, that is, candidates for original images.

The candidates for the original image in FIG. 2A are taken in the direction of 45 degrees to the right from the front left corner of the stadium F by the camera 11a installed around the stadium F (lower left corner in FIG. 1). This is a captured image.
The candidate of the original image in FIG. 2B is an image picked up in the direction from the front center line of the stadium F to the front by the camera 11b installed around the stadium F (lower center in FIG. 1). It is an image.
The candidate for the original image in FIG. 2 (c) is inclined 45 degrees to the left from the front right corner of the stadium F by the camera 11c installed around the stadium F (lower right corner in FIG. 1). It is a captured image.
The candidate for the original image in FIG. 2 (d) is captured in the direction of 45 ° diagonally right from the far left corner of the stadium F by the camera 11d installed around the stadium F (upper left corner in FIG. 1). This is a captured image.
The candidate for the original image in FIG. 2E is an image picked up in the direction from the back center line of the stadium F to the front by the camera 11e installed around the stadium F (upper center in FIG. 1). It is an image.
The candidate for the original image in FIG. 2 (f) is picked up in the direction of 45 degrees leftward from the far right corner of the stadium F by the camera 11f installed around the stadium F (upper right corner in FIG. 1). This is a captured image.
The candidate for the original image in FIG. 2G is a captured image captured in the direction from the back side of the goal G1 toward the goal G2 by the camera 11g installed around the stadium F (left corner in FIG. 1). It is.
The candidate for the original image in FIG. 2H is a captured image captured in the direction from the back side of the goal G2 toward the goal G1 by the camera 11h installed around the stadium F (right corner in FIG. 1). It is.

  In this embodiment, the captured image data of each of the cameras 11a to 11h becomes the original image candidate data as described above, and is selected from the candidate data based on the position information of the selected object OB. One is used as data of the original image.

Here, as a method for acquiring position information from the selected object OB, the following method is employed in the present embodiment.
That is, the sensors OB1 to OB7 that can be the selected object OB are equipped with sensors capable of acquiring position information, and the position information is sequentially transmitted as time passes. The method of acquiring is adopted.
Specifically, for example, the ball object OB1 includes a plurality of movement position sensor units as sensors capable of acquiring position information. Specifically, the movement position sensor unit includes a three-dimensional position measurement unit such as a three-axis acceleration sensor, a high-precision positioning unit, a wireless transmission unit that transmits three-dimensional position information to the external terminal 14 over time, It comprises power supply means for supplying power to positioning means and wireless transmission means.
Therefore, in this method, the image processing apparatus 12 acquires the position information by receiving the position information transmitted in a form such as a wireless LAN or Bluetooth (registered trademark) from the wireless transmission unit of the movement position sensor unit. To do.

When the image processing apparatus 12 acquires the position information of the selected object OB in this way, it is necessary for generating viewpoint-converted image data from the captured image data of the cameras 11a to 11h based on the position information. Select the original image data.
Specifically, the image processing apparatus 12 acquires data of captured images captured by each of the cameras 11a to 11h in the same time period as the time period when the position information of the selected object OB is acquired. The image processing apparatus 12 selects one of the captured image data to be used as the original image data based on the position information of the selected object.

The image processing apparatus 12 generates viewpoint-converted image data representing the view direction of the selected object OB moving in a predetermined movement direction from the data of one original image selected in this way.
For example, the image processing apparatus 12 can generate viewpoint-converted image data indicating that the virtual viewpoint is viewed with the same direction as the moving direction of the selected object as the viewing direction.
Further, the image processing apparatus 12 can generate viewpoint-converted image data having a size corresponding to the moving speed of the selected object. For example, when the moving speed of the selected object is high, the image processing device 12 generates wide-angle viewpoint-converted image data different from normal viewpoint-converted image data.

  FIG. 3 shows a specific example of the viewpoint conversion image.

In the example of FIG. 3, among the data of a plurality of captured images captured by each of the plurality of cameras 11 a to 11 h, that is, among the data of the plurality of captured images shown in each of FIGS. 2 (a) to 2 (h). The captured image data shown in FIG. 2H is selected as the original image data.
Then, the captured image data shown in FIG. 2H is converted based on a predetermined method, thereby generating the viewpoint-converted image data shown in FIG.

In this case, the predetermined method applied to the conversion of the viewpoint-converted image into data is not particularly limited, but in the present embodiment, for example, it can be realized by using a method using affine transformation.
By applying such a method using affine transformation, various objects OB and the like can be translated and rotated without changing the shape, or the magnification of enlargement or reduction can be changed between the vertical and horizontal directions. It becomes possible.
As a technique using affine transformation, any technique known or will appear in the future, for example, a technique disclosed in Japanese Patent Application Laid-Open No. 2007-133844 can be adopted. Such a method is adopted.

Specifically, in the affine transformation, a transformation matrix that transforms the coordinates of the original image from local coordinates to world coordinates is used. At this time, in the affine transformation, the object is enlarged / reduced / rotated / translated, etc. when moving from the local coordinates to the world coordinates. The “local coordinates” are coordinates representing the vertex information of an object in a coordinate system in which the center position of each object is (X, Y, Z) = (0, 0, 0).
The “world coordinates” are coordinates expressed in a coordinate system when individual objects are arranged in a common three-dimensional virtual space based on a relative positional relationship. In this “world coordinates”, not only objects but also light sources and viewpoints are arranged.
Next, in the affine transformation, a transformation matrix that converts world coordinates to view coordinates is used, and a transformation matrix that converts view coordinates to perspective coordinates is used. As a result, the three-dimensional information is converted into two-dimensional information that can be displayed on the display unit of the display device. At this time, the coordinate system is such that the center of display is (X, Y) = (0, 0). The “view (viewpoint) coordinates” are coordinates when the viewpoint direction is viewed from the viewpoint position in the coordinate system of “world coordinates”, that is, the viewpoint position is (X, Y, Z) = (0, 0, 0). ) In the coordinate system in which the viewpoint direction is the positive direction of the Z axis. Further, “perspective coordinates” are coordinates that represent three-dimensional information in a two-dimensional coordinate system. The “perspective coordinates” are also referred to as “screen coordinates”.
Finally, in affine transformation, a transformation matrix that transforms perspective coordinates into display coordinates is used. “Display coordinates” are coordinates expressed in a coordinate system corresponding to the size of the drawing area in the display unit. In “display coordinates”, the center of display is set to (X, Y) = (0, 0) in perspective coordinates, but the upper left point of the drawing area is set to (X, Y) so as to correspond to the drawing area in the display unit. = (0,0) conversion is performed.
By using such an affine transformation, the image processing device 12 can convert the data of the original image into data of a viewpoint-converted image in which the viewing direction of the selected object OB that moves in a predetermined movement direction is a virtual viewpoint. it can.

  Returning to FIG. 1, the relay device 13 relays various types of information exchanged between the image processing device 12 connected via the network N and the external terminal 14, for example, data of a viewpoint conversion image.

The external terminal 14 is configured by, for example, a smartphone or the like, and displays the viewpoint-converted image on a display or the like when receiving the viewpoint-converted image data transmitted from the image processing device 12 via the relay device 13, for example.
Here, if the network N is the Internet, the external terminal 14 can issue an instruction such as an acquisition request for viewpoint-converted image data by a URL (Uniform Resource Locator) request using, for example, HTTP (Hypertext Transfer Protocol). .
Further, the external terminal 14 can execute remote control such as an instruction to change the selected object by transmitting various control signals to the image processing device 12 via the relay device 13.

Next, the configuration of the image processing apparatus 12 as a main component in the image processing system 10 of the present embodiment will be described.
[Image processing device]
FIG. 4 is a block diagram showing a hardware configuration of the image processing apparatus 12 according to the present embodiment.

  The image processing apparatus 12 includes a CPU (Central Processing Unit) 31, a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, an image processing unit 34, a bus 35, an input / output interface 36, and an input. A unit 37, a display unit 38, a time measuring unit 39, a storage unit 40, a communication unit 41, and a drive 42 are provided.

  The CPU 31 executes various processes according to a program recorded in the ROM 32 or a program loaded from the storage unit 40 to the RAM 33.

  The RAM 33 appropriately stores data necessary for the CPU 31 to execute various processes.

  The image processing unit 34 includes a DSP (Digital Signal Processor), a VRAM (Video Random Access Memory), and the like, and performs various image processing on image data in cooperation with the CPU 31.

  The CPU 31, ROM 32, RAM 33, and image processing unit 34 are connected to each other via a bus 35. An input / output interface 36 is also connected to the bus 35. An input unit 37, a display unit 38, a timer unit 39, a storage unit 40, a communication unit 41, and a drive 42 are connected to the input / output interface 36.

The input unit 37 is configured with a keyboard, a mouse, and the like, and inputs various types of information in accordance with user instruction operations.
The display unit 38 includes a display and displays various images, for example, the above-described viewpoint conversion image.
The clock unit 39 performs a clock operation and acquires the current time.
The storage unit 40 is configured by a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various images, for example, the above-described viewpoint-converted image data.
The communication unit 41 communicates with other devices such as the object OB (see FIG. 1), the camera 11 (see FIG. 1), and the external terminal 14 (see FIG. 1) in the image processing system 10 via the network N including the Internet. Control communications between them.

  A removable medium 51 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 42. The program read from the removable medium 51 by the drive 42 is installed in the storage unit 40 as necessary. The removable medium 51 can also store various data such as image data stored in the storage unit 40 in the same manner as the storage unit 40.

FIG. 5 is a functional block diagram showing a functional configuration for executing the viewpoint-converted image generation processing among the functional configurations of the image processing apparatus 12 including such components.
The viewpoint converted image generation processing in this specification refers to the following series of processing. That is, the image processing apparatus 12 selects one of a plurality of captured image data transmitted from the plurality of cameras 11 as original image data from the position information of the selected object OB. The image processing device 12 performs image conversion processing on the original image data, thereby generating viewpoint-converted image data indicating that the virtual viewpoint views the viewing direction of the selected object OB.

  Here, in order to facilitate understanding of the functional configuration of the image processing apparatus 12, the camera 11 and the object OB will be described before detailed description thereof.

  As shown in FIG. 5, the camera 11 includes an imaging unit 101 and a transmission unit 102.

  Although not shown, the imaging unit 101 includes an optical lens unit and an image sensor.

The optical lens unit is configured by a lens that collects light, for example, a focus lens or a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. A digital signal is generated by various signal processing and output as an output signal of the imaging unit 101.
Such an output signal of the imaging unit 101 is appropriately supplied to the transmission unit 102. That is, the imaging unit 101 continuously shoots the surrounding space, and temporarily stores the moving image, still image, or continuous-shot image data obtained as a result thereof, that is, the above-described captured image data in a buffer (not shown). In turn, compression encoding is performed.
The transmission unit 102 wirelessly transmits the captured image data output from the imaging unit 101 to the image processing apparatus 12.

A sensor unit 111 and a transmission unit 112 are attached to the object OB.
The sensor unit 111 is configured as a three-axis acceleration sensor in the present embodiment, and is configured by a three-axis acceleration sensor including a piezoresistive type or a capacitance type detection mechanism. X, Y, Z) components are detected, and data indicating the detection results are output. The data indicating the detection result of the sensor unit 111 is hereinafter referred to as “position information data”. The position information data is appropriately supplied to the transmission unit 112.
Here, in the position information data, the X component is the vertical vibration cycle of the object OB, the Y component is the horizontal vibration cycle of the object OB, and the Z component is the vibration cycle of the object OB in the traveling direction. Correspond.
Further, the sensor unit 111 can output triaxial acceleration data corrected according to the tilt angle even in an arbitrary posture (tilt angle) state. Therefore, the sensor unit 111 corrects data output from an inclination sensor (not shown) using a gyro sensor or the like in accordance with the triaxial acceleration data corrected according to the inclination angle. As a result, when the object OB is used in a moving state in which an external force such as a centrifugal force is applied, for example, when a subject is imaged during movement, accurate acquisition of various data and accurate positioning calculation can be performed. Become.

The transmission unit 112 wirelessly transmits the position information data output from the sensor unit 111 to the image processing apparatus 12.
In the present embodiment, the sensor unit 111 and the transmission unit 112 are unitized as the movement position sensor unit described above, and are built in the ball object OB1 or attached to the player objects OB2 to OB7.

When the viewpoint conversion image generation process is executed in a state where such an object OB exists, as shown in FIG. 5, in the CPU 31 of the image processing apparatus 12, the main control unit 61, the captured image acquisition unit 62, The position information acquisition unit 63, the storage control unit 64, the display control unit 65, and the communication control unit 66 function. In the image processing unit 34, a moving direction calculating unit 71, a moving speed calculating unit 72, an original image selecting unit 73, and a viewpoint conversion image processing unit 74 function.
However, FIG. 5 is an example, and at least a part of the function of the image processing unit 34 may be delegated to the CPU 31, and conversely, at least a part of the function of the CPU 31 may be delegated to the image processing unit 34. .
Further, an original image candidate data storage unit 81, an object data storage unit 82, and a display data storage unit 83 are provided as one area of the storage unit 40 of the image processing device 12. Note that the original image candidate data storage unit 81, the object data storage unit 82, and the display data storage unit 83 are provided as one area of the storage unit 40, and other examples are provided as one area of the removable media 51, for example. You may be made to do.

  The original image candidate data storage unit 81 sequentially stores data of captured images that are sequentially transmitted from the camera 11 and received by the image processing device 12 as candidates for the original image. The captured image data is stored in the original image candidate data storage unit 81 together with the acquired time.

  The object data storage unit 82 stores position information data sequentially transmitted from the object OB and received by the image processing apparatus 12. The position information data is stored in the object data storage unit 82 together with the acquired time.

  The display data storage unit 83 stores data of a viewpoint converted image generated by a viewpoint converted image processing unit 74 (to be described later) of the image processing unit 34 to be displayed on the display unit 38.

  The main control unit 61 controls execution of various processes in each unit of the image processing apparatus 12. Specifically, the main control unit 61 receives various instructions from the input unit 37 and the external terminal 14 and performs various controls such as setting of the selected object and control of imaging in the camera 11.

  The captured image acquisition unit 62 acquires captured image data transmitted from the cameras 11a to 11h via the communication unit 41, and the captured image data is time data acquired from the time measuring unit 39 as supplementary information. In addition, the control for storing the original image candidate data in the original image data storage unit 81 is executed.

  The position information acquisition unit 63 acquires the position information data transmitted from the objects OB <b> 1 to OB <b> 7 via the communication unit 41. Thereafter, the position information acquisition unit 63 supplies the position control data of the objects OB1 to OB7 to the storage control unit 64.

  The storage control unit 64 executes control for storing various data in the storage unit 40. Specifically, the storage control unit 64 stores the position information data acquired by the position information acquisition unit 63 in the object data storage unit 82 by adding the time data acquired from the time measuring unit 39 as supplementary information. Execute control.

  The display control unit 65 performs control to display and output various images on the display unit 38. Specifically, the display control unit 65 executes control for causing the display unit 38 to display the viewpoint conversion image generated by the image processing unit 34.

  The communication control unit 66 executes control for performing wireless communication with the camera 11 and the external terminal 14. Specifically, the communication control unit 66 is connected to the camera 11 and the external terminal 14 via the communication unit 41 and the relay device 17 that relays the network N, and executes control for transferring various information.

  The movement direction calculation unit 71 calculates the movement direction of the selected object based on the position information data stored in the object data storage unit 82. That is, the movement direction calculation unit 71 calculates the movement direction by determining the positional relationship between the position information data from the history of the position information data stored in the object data storage unit 82.

  The moving speed calculation unit 72 calculates the moving speed of the selected object OB based on the position information data stored in the object data storage unit 82. That is, the movement direction calculation unit 71 calculates the movement speed by calculating the movement distance between the pieces of position information data from the history of the position information data stored in the object data storage unit 82.

  The original image selection unit 73 is stored in the original image candidate data storage unit 81 based on the position information stored in the object data storage unit 82 and the moving direction and moving speed calculated by the moving speed calculation unit 72. , One predetermined image is selected as the original image data from the captured image data captured at the same time. Specifically, the original image selection unit 73 captures an image in which the moving direction of the object whose virtual viewpoint is variably arranged among the objects OB1 to OB7 and the direction of the optical axis of the camera 11 (actual viewpoint) are most approximated. Are selected as original image data.

The viewpoint conversion image processing unit 74 processes the viewpoint conversion by converting the original image data, and the virtual viewpoint variably arranged on the selected object OB changes the moving direction of the selected object OB from the viewing direction. Data of a viewpoint conversion image representing a state of viewing is generated.
Specifically, the viewpoint conversion image processing unit 74 is configured to select the original image selected by the original image selection unit 73 based on the movement direction calculated by the movement direction calculation unit 71 and the movement speed calculated by the movement speed calculation unit 72. The image data is converted to generate viewpoint-converted image data.
In the present embodiment, the viewpoint conversion image processing unit 74 variably arranges the data of one captured image selected by the original image selection unit 73 in the selected object OB by the above-described affine transformation. The virtual viewpoint in which the virtual viewpoint is variably arranged on the selected object OB generates point-converted image data representing a state in which the moving direction of the selected object OB is viewed from the viewing direction.

Heretofore, the functional configuration of the image processing apparatus 12 according to the present embodiment has been described. Subsequently, a viewpoint converted image generation process executed by the image processing apparatus 12 having such a functional configuration will be described with reference to FIG.
FIG. 6 is a flowchart for explaining the flow of viewpoint-converted image generation processing executed by the image processing apparatus 12 having the functional configuration of FIG.

A user who operates the input unit 37 of the image processing apparatus 12 can perform a predetermined operation instructing the start of the viewpoint conversion image generation process.
The viewpoint-converted image generation process is started when such a user performs a predetermined operation on the input unit 37, and the following process is executed.

  In step S11, the main control unit 61 performs object setting (initial setting). That is, the main control unit 61 performs a predetermined operation on the input unit 37 by the user, and sets the ball object OB1 among the ball object OB1 and the player objects OB2 to OB7 shown in FIG. And set as the selected object OB.

  In step S12, the captured image acquisition unit 62 acquires data of a plurality of captured images output from a plurality of cameras. That is, as shown in FIG. 1, the captured image acquisition unit 62 captures images captured by a plurality of cameras 11 a to 11 h that are capturing the stadium F including the ball object OB1 and the player objects OB2 to OB7 from the surroundings. Image data is acquired via the communication unit 41. The captured image acquisition unit 62 acquires captured image data as illustrated in FIGS. 2A to 2H in which the stadium F is captured from different directions by a plurality of cameras 11a to 11h, for example. Thereafter, the captured image acquisition unit 62 stores the acquired captured image data in the original image candidate data storage unit 81 by adding the time data acquired from the time measuring unit 39 as supplementary information.

  In step S <b> 13, the position information acquisition unit 63 acquires position information data of the selected object OB. That is, the position information acquisition unit 63 acquires data of position information measured by the sensor unit 111 of the selected object OB via the communication unit 41.

  In step S14, the storage control unit 64 stores the position information of the selected object OB and the time in association with each other. That is, the storage control unit 64 associates the current time measured by the time measuring unit 39 with the position information acquired by the position information acquisition unit 63 and causes the object data storage unit 82 to store it.

  In step S15, the moving direction calculation unit 71 calculates the moving direction of the selected object OB from the history of the position information data of the selected object OB. That is, the movement direction calculation unit 71 acquires the history of the position information data of the selected object OB stored in the object data storage unit 82, and calculates the movement direction based on the transition of each position.

  In step S16, the moving speed calculation unit 72 calculates the moving speed of the selected object OB from the history of the position information data of the selected object OB. That is, the moving speed calculation unit 72 acquires the history of the position information data of the selected object OB stored in the object data storage unit 82, and calculates the moving distance per unit time as the moving speed.

  In step S <b> 17, the original image selection unit 73 selects original image data from a plurality of captured image data based on the position, moving direction, and moving speed of the selected object OB. That is, the original image selection unit 73 selects captured image data optimal for generating viewpoint-converted image data as original image data.

  In step S <b> 18, the viewpoint conversion image processing unit 74 generates viewpoint conversion image data based on the selected original image data. That is, the viewpoint conversion image processing unit 74 performs image conversion (affine transformation in this embodiment) on the data of the original image selected by the original image selection unit 73, so that the virtual viewpoint variably arranged on the selected object becomes While moving in the moving direction of the selected object, data of a viewpoint conversion image representing a state in which the moving direction is viewed as the viewing direction is generated. The data of the viewpoint conversion image is stored in the display data storage unit 83 under the control of the storage control unit 64.

  In step S19, the display control unit 65 executes control for causing the display unit 38 to display and output the viewpoint-converted image data generated in the process of step S18. That is, the viewpoint conversion image is displayed on the display unit 38.

  In step S <b> 20, the communication control unit 66 transmits the viewpoint conversion image data to the external terminal 14 via the communication unit 41.

In step S21, the main control unit 61 determines whether or not an operation for instructing to change the selected object (hereinafter referred to as “selected object change instruction operation”) has been received.
If the selected object change instruction operation is accepted, it is determined as YES in Step S21, and the process proceeds to Step S22.

  In step S22, the main control unit 61 changes the selected object OB to another object OB selected from the plurality of objects OB2 to OB7. When this process ends, the process proceeds to step S23.

  On the other hand, when the selected object change instruction operation is not accepted, it is determined as NO in Step S21, and the process proceeds to Step S23.

In step S23, the main control unit 61 determines whether or not an instruction to end the viewpoint conversion image generation process has been received.
If the end instruction has not been received, it is determined as NO in Step S23, and the process returns to Step S12. That is, the processing from step S12 to step S23 is repeatedly executed until an instruction to end the viewpoint conversion image generation processing is received. Then, when an instruction to end the viewpoint-converted image generation process is accepted, it is determined as YES in Step S23, and the viewpoint-converted image generation process ends.

[First Modification of First Embodiment]
In the first embodiment described above, an example in which the selected object is a ball has been described. However, in this modification, an example in which a player is used as a selected object instead of the ball will be described with reference to FIGS. 7 and 8. explain. Note that the same parts as those in the above-described embodiment are denoted by the same reference numerals, detailed description thereof is omitted, and only different portions are described.

FIG. 7 is a schematic diagram showing a schematic configuration of the image processing system according to the first modified example of the first embodiment of the present invention, and shows a case where the player objects OB2 to OB7 are selected as viewpoints. is there.
As shown in FIG. 7, the image processing apparatus 12 sets the player's selection object OB3 among the objects OB1 to OB7 as a virtual viewpoint, that is, as a processing object using the virtual viewpoint, as a selection object OB. You can choose.

  The image processing apparatus 12 is moving in the moving direction A2 with at least one original image data of the captured image data as shown in FIGS. 2A to 2H captured by the cameras 11a to 11h. A series of viewpoint conversion images (moving images, continuous shot images, or images) showing a view from the virtual viewpoint in the viewing direction of the selection object OB3 moving in the movement direction A2 based on the position information sequentially transmitted from the selection object OB3. Still image) data.

FIG. 8 shows another specific example of the viewpoint conversion image generated by the image processing device 12.
As shown in FIG. 8, since the selected object OB3 is a player, the image processing device 12 displays a series of viewpoint conversion images that are generated as viewed from the height of the player's line of sight as viewed in the moving direction A2. Generate image data. Therefore, the image processing apparatus 12 moves the image data representing the state in which the stadium F is viewed from the outside (from the outside) from the actual viewpoint of the cameras 11a to 11h in the moving direction A2 inside the stadium F. The data of the viewpoint conversion image is generated by converting the object OB3 into image data representing a view from the virtual viewpoint in the viewing direction (virtual viewpoint serving as the player's line of sight).

[Second Modification of First Embodiment]
In the first embodiment described above, an example in which the image processing apparatus 12 acquires position information transmitted from a moving position sensor unit equipped with a three-axis acceleration sensor will be described as a method for acquiring position information from the selected object OB. However, this modification demonstrates the example which acquires the positional information transmitted from a movement position sensor unit with the receiver arrange | positioned in the same position as the camera 11, or a different position. Note that the same parts as those in the above-described embodiment are denoted by the same reference numerals, detailed description thereof is omitted, and only different portions are described.

FIG. 9 is a schematic diagram illustrating an example of a method for acquiring position information of the selected object OB.
In the example shown in FIG. 9, the receiving device 15 is installed in the vicinity of the camera 11. The receiving device 15 receives position information data transmitted from a movement position sensor unit built in the ball by a wireless LAN, Bluetooth (registered trademark) or the like, and transmits the data to the image processing device 12.
In order to cover the entire range of the stadium F, the receiving device 15 is provided with three receiving devices 15-1 to 15-3 corresponding to the cameras 11a to 11c.
In the image processing system configured as described above, the position information data of the selected object OB measured by the sensor unit of the movement position sensor unit is transmitted from the transmission unit of the movement position sensor unit to the reception device 15. Thereafter, the image processing device 12 uses the position information data received from the receiving device 15 together with the captured image data acquired from the camera 11 to generate viewpoint-converted image data.
In this modification, the receivers 15-1 to 15-3 are arranged in the vicinity of the camera 11, that is, in the same position as the cameras 11a to 11c. However, the present invention is not limited to this, and the camera is not related to the position of the camera 11. 11 may be configured so that the receiving device is arranged at a position different from 11.
[Third Modification of First Embodiment]
In the first embodiment described above, the example in which the position information from the triaxial acceleration sensor is used as the method for acquiring the position information from the selected object OB has been described. However, in the present modification, a positioning sensor is not provided for the object. The position information from the selected object OB is acquired using the electromagnetic field. Note that the same parts as those in the above-described embodiment are denoted by the same reference numerals, detailed description thereof is omitted, and only different portions are described.
FIG. 10 is a schematic diagram illustrating an example of a method for acquiring position information of the selected object OB.
In the example shown in FIG. 10, a magnetic field generation source 16 and a reception unit 17 are provided.
The magnetic field generation source 16 is composed of an antenna coil or the like, and generates an electromagnetic field. In order to cover the entire range of the stadium F, the magnetic field generation source 16 is provided with three magnetic field generation sources 16-1 to 16-3 corresponding to the cameras 11a to 11c.
The receiving unit 17 is configured by a “hippo mass sensor” or the like that receives and detects an electromagnetic field generated by the magnetic field generation source 16 and measures a three-dimensional position from a combination of received magnetic strengths.
The receiving means 17 is provided with three receiving means 17-1 to 17-3 in the cameras 11d to 11f corresponding to the magnetic field generating sources 16-1 to 16-3.
In the image processing system configured as described above, an electromagnetic field is generated by the magnetic field generation sources 16-1 to 16-3, and a combination of reception strengths of the electromagnetic fields received by the receiving units 17-1 to 17-3 is used. The three-dimensional position of the selected object OB is measured.

[Fourth Modification of First Embodiment]
In the first embodiment described above, the example in which the position information from the triaxial acceleration sensor is used as the method for acquiring the position information from the selected object OB has been described. However, in the present modification, a positioning sensor is not provided for the object. The position information from the selected object OB is acquired using the Doppler effect.

The image processing system 10 includes a beacon radio antenna.
A plurality of beacon radio antennas are provided corresponding to the cameras, and transmit radio radio waves having different frequencies.

  The image processing system 10 configured as described above transmits radio waves from a plurality of beacon radio antennas having different frequencies. The image processing apparatus 12 includes a so-called “radio wave Doppler sensor” that calculates a three-dimensional position and a relative position of a selected object from a frequency transition received from a beacon radio antenna.

  In this modification, the receiving device is arranged near the camera 11, that is, at the same position as the camera 11. However, the present invention is not limited to this, and the receiving device is arranged at a position different from the camera 11 regardless of the position of the camera 11. You may comprise.

[Fifth Modification of First Embodiment]
In the first embodiment described above, the example in which the position information from the triaxial acceleration sensor is used as the method for acquiring the position information from the selected object OB has been described. However, in this modification, the object OB is used for positioning. Without providing a sensor, the picked-up image acquired from the camera 11 is subjected to image processing to identify the selected object and acquire position information.

FIG. 11 is a schematic diagram illustrating an example of a method for acquiring position information of the selected object OB.
Specifically, as shown in FIG. 11, the position information of the selected object OB is obtained by attaching a plurality of specific marks M1 to M4 (for example, stars) on the surface of the selected object such as a ball, and the mark M1. -M4 detects the ball from the captured image and marks M1 to M4 on the surface of the selected object in the continuously captured images (hereinafter, if there is no need to individually distinguish the marks M1 to M4) Collectively, the relative position of “mark M” and the change between the captured images may be calculated from the movement position to calculate the three-dimensional position and the relative position of the selected object.
In this modification, the selected object OB is obtained from the mark M serving as identification information. However, the present invention is not limited to this, and the position information can be calculated by specifying the appearance of the object. . That is, when the ball is the selected object OB, the shape of the ball is specified from the captured image, and the position information data of the ball is acquired by determining the position of the specified ball from the captured image.
[Other Modifications of First Embodiment]
In the first embodiment described above, the example in which the position information from the three-axis acceleration sensor is used as the method for acquiring the position information from the selected object OB has been described. Various 3D measurement methods such as lens focus method, passive or active triangulation stereo method, optical laser method, illuminance difference stereo method, slit light projection method, other electromagnetic field type, wireless type 3D measurement device, etc. A measurement method may be adopted.

As described above, the image processing apparatus 12 according to this embodiment includes the captured image acquisition unit 62, the position information acquisition unit 63, the viewpoint conversion image processing unit 74, and the display control unit 65.
The captured image acquisition unit 62 acquires data of a plurality of captured images captured by each of the plurality of cameras 11 that capture a predetermined space in which the object OB can move from different positions.
The position information acquisition unit 63 acquires the position of the object OB moving in a predetermined space.
The viewpoint conversion image processing unit 74 processes the data of a plurality of captured images acquired by the captured image acquisition unit 62 based on the position acquired by the position information acquisition unit 63, thereby processing the object OB. Data of an image representing a state in which the arranged virtual viewpoint looks at a predetermined viewing direction is generated as viewpoint converted image data.
The display control unit 65 performs control to display and output the viewpoint converted image data generated by the viewpoint converted image processing unit 74.

The image processing apparatus 12 performs viewpoint conversion on the captured image data acquired by the captured image acquisition unit 62 based on the position acquired by the position information acquisition unit 63, thereby performing virtual conversion on the object OB. Data of an image representing a state in which the viewpoint looks at a predetermined viewing direction is generated as viewpoint converted image data.
In this case, the image processing device 12 can easily obtain a viewpoint conversion image equivalent to that captured from the viewpoint desired by the viewer.
Further, the image processing apparatus 12 displays the generated viewpoint-converted image data, so that the moving object OB can be used as a viewpoint without providing an imaging device such as a camera to the moving selection object OB. An image full of feeling can be obtained.
The image processing apparatus 12 can obtain a high-quality image because it is not necessary to provide an imaging unit for the moving object OB and is not restricted by imaging.

The position information acquisition unit 63 acquires the position of the object OB moving in a predetermined space at an arbitrary timing.
The viewpoint conversion image processing unit 74 processes the data of the captured image acquired by the captured image acquisition unit 62 based on the position of the object acquired by the position information acquisition unit 63 at an arbitrary timing.
In this case, in order to process the captured image based on the position of the arbitrary timing acquired by the position information acquisition unit 63, for example, the image processing device 12 converts the viewpoint-converted image at the position of the arbitrary timing designated by the user. Data can be generated.

The position information acquisition unit 63 sequentially acquires the position of the object OB moving in a predetermined space.
The viewpoint conversion image processing unit 74 sequentially processes captured image data sequentially acquired by the captured image acquisition unit 62.
In this case, the image processing device 12 is not an image stored in advance (recorded image) in order to sequentially process the data of the captured image based on the position of the object OB moving in the predetermined space acquired sequentially. The captured image data acquired in real time can be processed to generate viewpoint converted image data.

Further, the image processing apparatus 12 is configured to generate an original image that is a source of generation of an image representing a state viewed from a virtual viewpoint arranged on the object from a plurality of captured image data acquired by the captured image acquisition unit 62. An original image selection unit 73 that selects data based on the position acquired by the position information acquisition unit 63.
The viewpoint conversion image processing unit 74 performs viewpoint conversion processing on the data of the original image acquired by the original image selection unit 73, so that the virtual viewpoint arranged on the object looks at a predetermined viewing direction. Are generated as viewpoint-converted image data.
In this case, the image processing apparatus 12 generates a plurality of imaging images in order to generate viewpoint-converted image data based on the original image data selected from the plurality of captured image data by the original image selection unit 73. If picked-up image data suitable for generating viewpoint-converted image data is selected from the image data, it is possible to reduce the processing load of the processing process and to generate highly accurate viewpoint-converted image data.

Further, the image processing apparatus 12 includes a movement direction calculation unit 71 that calculates the movement direction of the object OB.
The original image selection unit 73 selects original image data based on the movement direction of the object OB acquired by the movement direction calculation unit 71.
The viewpoint conversion image processing unit 74 selects original image data based on the movement direction of the object OB calculated by the movement direction calculation unit 71, and processes the original image data based on the movement direction of the object OB. Then, data of an image representing a state in which the virtual viewpoint of the object OB is viewed with the moving direction as the viewing direction is generated as viewpoint converted image data.

  In this case, the image processing apparatus 12 processes the data of the original image based on the movement direction calculated by the movement direction calculation unit 71, so that the virtual viewpoint arranged in the object OB moving in the movement direction changes the movement direction. By generating the image data representing the view as the viewing direction as the viewpoint conversion image data, an image with higher presence can be obtained.

Further, the image processing apparatus 12 includes a movement speed calculation unit 72 that calculates the movement speed of the object OB.
The viewpoint conversion image processing unit 74 generates image data of an angle of view corresponding to the movement speed acquired by the movement speed calculation unit 72 as viewpoint conversion image data.

  In this case, since the image processing device 12 can generate image data of an angle of view corresponding to the moving speed acquired by the moving speed calculation unit 72 as the viewpoint conversion image data, a sense of speed is further added. A highly realistic image can be obtained.

  The viewpoint conversion image processing unit 74 generates data of a wide angle image as the viewpoint conversion image data in inverse proportion to the moving speed. For this reason, the image processing device 12 can obtain a viewpoint-converted image with a natural sense of speed.

Further, the object OB acquires and transmits data indicating its own state.
The position information acquisition unit 63, the movement direction calculation unit 71, or the movement speed calculation unit 72 acquires data transmitted from the object OB, and acquires or calculates the position, movement direction, or movement speed of the object based on the data. .
In this case, the image processing apparatus 12 can acquire the position, moving direction, or moving speed of the object OB by receiving the position information of the moving object OB. The position, moving direction of the object OB can be obtained with a simple configuration. Alternatively, the moving speed can be acquired.

In addition, the image processing device 12 is provided with a receiving device that receives data indicating the position, moving direction, or moving speed of the object OB transmitted from the object OB.
The position information acquisition unit 63, the movement direction calculation unit 71, or the movement speed calculation unit 72 acquires data indicating the position, movement direction, or movement speed of the object OB via the reception device.
In this case, since the image processing apparatus 12 can acquire data indicating the position, moving direction, or moving speed of the object OB via the receiving apparatus, the position, moving direction, or moving speed of the object OB with high accuracy can be obtained. Can be acquired.

Further, the image processing apparatus 12 is provided with a generation source that generates an electromagnetic field and a receiving unit that receives and detects the electromagnetic field.
The position information acquisition unit 63, the movement direction calculation unit 71, or the movement speed calculation unit 72 acquires the position, movement direction, or movement speed of the object OB based on the reception detection result of the reception unit.
In addition, the image processing apparatus 12 is provided with a beacon radio wave antenna that transmits radio waves having different frequencies.
The position information acquisition unit 63, the movement direction calculation unit 71, or the movement speed calculation unit 72 acquires the position, movement direction, or movement speed of the object OB by using the Doppler effect by the radio wave transmitted from the beacon radio wave antenna. .
Therefore, the image processing apparatus 12 does not include a device for individually acquiring position information on the object OB by acquiring the position, moving direction, or moving speed of the object OB as described above. That is, it is not necessary to provide the object OB with a configuration related to acquisition of position information.

The position information acquisition unit 63, the movement direction calculation unit 71, or the movement speed calculation unit 72 performs image processing on the captured image data in which the object OB is captured among the plurality of captured image data acquired by the captured image acquisition unit 62. And the position, moving direction, or moving speed of the object OB is acquired based on the result of the image processing.
In this case, since the image processing apparatus 12 can acquire the position, moving direction, or moving speed of the object OB from the captured image that has been subjected to image processing, the object OB is not mounted on the object OB without mounting a separate device or the like. Position, moving direction, or moving speed can be acquired.

Further, identification information (mark M) is attached to the object OB.
The position information acquisition unit 63, the movement direction calculation unit 71, or the movement speed calculation unit 72 recognizes the identification information (mark M) by image processing, and uses the identification information (mark M) to detect the position and movement direction of the object OB. Alternatively, the moving speed is acquired.
In this case, since the image processing apparatus 12 can acquire the position, moving direction, or moving speed of the object OB from the mark M serving as identification information written on the object OB, a separate device or the like is mounted on the object OB. In addition, the position, moving direction, or moving speed of the object OB can be acquired by light image processing.

In addition, the position information acquisition unit 63, the movement direction calculation unit 71, or the movement speed calculation unit 72 recognizes the appearance of the object OB through image processing, and based on the appearance, the state (position, movement direction, or movement speed) of the object OB. ) To get.
In this case, the image processing apparatus 12 can acquire the position, moving direction, or moving speed of the object OB by recognizing and detecting the appearance of the object OB. In addition, the position, moving direction, or moving speed of the object OB can be acquired by light image processing.

In the image processing device 12, a plurality of objects exist in a space captured by the plurality of cameras 11.
The position information acquisition unit 63, the movement direction calculation unit 71, or the movement speed calculation unit 72 acquires the position, movement direction, or movement speed of each of the plurality of objects.
The viewpoint conversion image processing unit 74 includes a position of a selected object among the positions, movement directions, or movement speeds of the plurality of objects acquired by the position information acquisition unit 63, the movement direction calculation unit 71, or the movement speed calculation unit 72, Based on the moving direction or moving speed, data of the viewpoint conversion image is generated.
In this case, the image processing apparatus 12 generates viewpoint-converted image data based on the position, moving direction, or moving speed of the selected object among the positions, moving directions, or moving speeds of the plurality of objects OB. By doing so, it is possible to obtain an image that suits the taste.

In addition, the display control unit 65 performs control to display and output viewpoint conversion image data.
In this case, the image processing apparatus 12 can confirm the viewpoint conversion image generated by the image processing apparatus by executing the control to display and output the data of the viewpoint conversion image.

In addition, the display control unit 65 functions as a transmission unit that transmits the viewpoint converted image data generated by the viewpoint converted image processing unit 74 to an external device.
In this case, the viewpoint conversion image generated by the external terminal 14 other than the image processing apparatus can be confirmed, and the viewpoint conversion image can be enjoyed even at a distant place.

The image processing apparatus 12 further includes a control receiving unit that receives a control signal from the external terminal 14.
The viewpoint conversion image processing unit 74 generates, as viewpoint conversion image data, image data representing a state viewed from the virtual viewpoint indicated by the control signal received by the control reception unit.
In this case, the image processing apparatus 12 uses a viewpoint requested by the user of the external terminal 14 in order to generate image data representing a state viewed from the virtual viewpoint indicated by the control signal received by the control receiving unit. The viewpoint conversion image data can be generated.

The captured image acquisition unit 62 acquires moving image data as each of the plurality of captured image data.
The viewpoint conversion image processing unit 74 generates moving image data as viewpoint conversion image data based on the moving image data acquired by the captured image acquisition unit 62.
In this case, since the image processing apparatus 12 can provide a moving image with movement, it can provide a video with a more realistic feeling.
The viewpoint conversion image processing unit 74 is configured to generate moving image data as viewpoint conversion image data, but may be configured to generate continuous shot image data or still image data.

The image processing apparatus 12 according to the first embodiment of the present invention has been described above.
Next, an image processing apparatus 12 according to the second embodiment of the present invention will be described.

[Second Embodiment]
In the first embodiment described above, an example in which data of one captured image among the data of a plurality of captured images is used when generating viewpoint-converted image data. However, in the present embodiment, a plurality of captured images is used. An example of generating viewpoint-converted image data using this data will be described.
The image processing apparatus 12 according to the second embodiment of the present invention can have basically the same hardware configuration and functional configuration as the image processing apparatus 12 according to the first embodiment.
Therefore, FIG. 4 is also a block diagram showing a hardware configuration of the image processing apparatus 12 according to the second embodiment.
FIG. 5 is also a functional block diagram showing a functional configuration of the image processing apparatus 12 according to the second embodiment, but the original image selection unit 73 and the viewpoint conversion image processing unit 74 are different in the second embodiment.
That is, in the second embodiment, since viewpoint conversion image data is generated from data of a plurality of captured images, the original image selection unit 73 selects optimal captured image data from among the plurality of captured image data. Without selection, captured image data in which the selected object is reflected or captured image data suitable for generating viewpoint-converted image data is selected from the plurality of captured image data.
Further, the viewpoint conversion image processing unit 74 generates a viewpoint conversion image by converting from the difference between the data of the plurality of captured images or by converting and combining the data of the plurality of captured images. In this case, any known method can be adopted for generating the viewpoint conversion image. A known arbitrary technique is disclosed in, for example, Japanese Patent Application Laid-Open No. 2011-77982.

  FIG. 12 is a flowchart for explaining the flow of viewpoint-converted image generation processing according to the second embodiment, which is executed by the image processing device 12 of FIG. 4 having the functional configuration of FIG.

In the viewpoint-converted image generation process of FIG. 6 in the first embodiment described above, one captured image data serving as the original image data is selected from the plurality of captured image data, and the viewpoint is obtained by converting the original image data. In contrast to the generation of converted image data, the viewpoint-converted image generation processing of FIG. 12 in the second embodiment uses a plurality of captured image data to convert and synthesize each captured image data. Thus, the difference is that the data of the viewpoint conversion image is generated. Specifically, the second embodiment is different in that the process of step S45 is executed instead of the processes of steps S15 to S18 according to the first embodiment.
Thus, hereinafter, such differences will be mainly described, and description of the coincident points will be omitted as appropriate.

In step S45, the viewpoint conversion image processing unit 74 generates a viewpoint conversion image from a plurality of original image data based on the position information. In other words, the viewpoint conversion image processing unit 74 generates a viewpoint conversion image by converting from the difference between the data of the plurality of captured images or by converting and combining the data of the plurality of captured images.
At this time, the original image selection unit 73 may be configured to select in advance the captured image data suitable for the viewpoint-converted image data from among the plurality of captured image data as the original image.

  In the second embodiment described above, the position information acquisition method from the selected object OB of the first embodiment can be adopted, but the methods of the second to fifth modifications of the first embodiment are also possible. Can also be employed as appropriate.

In the image processing apparatus 12 configured as described above, the original image selection unit 73 selects a plurality of original image data from a plurality of captured image data captured by the plurality of cameras 11.
The viewpoint conversion image processing unit 74 is based on the data of the plurality of original images selected by the original image selection unit 73 (more specifically, the image data of the difference between the data of the plurality of original images). Generate data.

  In this case, the image processing apparatus 12 can obtain a more realistic viewpoint conversion image by generating the viewpoint conversion image data based on the data of the plurality of selected original images.

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

  In the above-described embodiment, the above-described embodiment to which the present invention is applied has been described with respect to an example in which a competition equipment such as a ball or a player is selected as a selection object. For example, a person in a competition venue such as a referee, a commentator, or a customer, or an object in a competition venue can be selected as a selection object.

  In the above-described embodiment, the imaging unit 101 transmits image data of continuously captured images to the image processing apparatus 12 through the sequential transmission unit 102, but is not limited thereto. For example, after being stored in a large-capacity memory (not shown), it can be divided and transmitted to the image processing apparatus 12 at predetermined time intervals, capacity intervals, or communication speeds.

  In the above-described embodiment, a plurality of cameras 11 are provided so that the entire area of the stadium F can be imaged. However, the present invention is not limited to this. For example, a wide-angle camera or a panoramic camera capable of imaging a wider range than the normal imaging range. It may be configured by an all-sky camera. In addition, the camera 11 may be configured by a plurality of wide-angle cameras, panoramic cameras, and omnidirectional cameras having different installation locations and viewing angles.

  In the above-described embodiment, the image processing system 10 in which the image processing apparatus 12 to which the present invention is applied has been described as an example used in a stadium where sports are performed. However, the present invention is not limited thereto. For example, the present invention can be applied to stage broadcasts.

  In the above-described embodiment, the selection object OB is set and the viewpoint-converted image data is generated according to the instruction from the image processing apparatus 12, but the present invention is not limited to this. For example, the selection object OB may be set according to an instruction from an external terminal to generate viewpoint converted image data.

  In the above-described embodiment, the viewpoint conversion image data is transmitted from the image processing apparatus 12 to the external terminal 14, but the present invention is not limited thereto. For example, the viewpoint-converted image data may be stored in a server connected to the network, and the viewpoint-converted image data may be acquired by an external terminal afterwards or in real time.

  In the above-described embodiment, the viewpoint conversion image data is sequentially generated from the acquired position information data. However, the present invention is not limited to this. For example, a predetermined time may be specified from the position information data history, and the viewpoint-converted image data in the specified time zone may be generated.

  In the above-described embodiment, for example, when the moving speed of the selected object is high, the image processing device 12 can change the selection object OB such as wide-angle viewpoint conversion image data different from normal viewpoint conversion image data. However, the present invention is not limited to this, and is configured to generate viewpoint-converted image data having a size corresponding to the moving speed. The image processing apparatus 12 may express the moving speed of the selected object by applying an effect such that the moving speed of the selected object is visually transmitted. For example, when the moving speed of the selected object is fast, the viewing direction It is also possible to generate viewpoint-converted image data that focuses on the vicinity of the center and blurs the surroundings.

In the above-described embodiment, the image processing apparatus 12 to which the present invention is applied has been described using a digital camera as an example, but is not particularly limited thereto.
For example, the present invention can be applied to general electronic devices having a viewpoint conversion image generation processing function. Specifically, for example, the present invention can be applied to a notebook personal computer, a printer, a television receiver, a video camera, a portable navigation device, a mobile phone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or can be executed by software.
In other words, the functional configuration of FIG. 5 is merely an example, and is not particularly limited. That is, it is sufficient that the image processing apparatus 12 has a function capable of executing the above-described series of processing as a whole, and what kind of functional block is used to realize this function is not particularly limited to the example of FIG. .
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.

When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only configured by the removable medium 51 of FIG. 4 distributed separately from the apparatus main body in order to provide the user with the program, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in. The removable medium 51 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preliminarily incorporated in the apparatus main body includes, for example, the ROM 32 in FIG. 4 in which a program is recorded, the hard disk included in the storage unit 40 in FIG.

In the present specification, the step of describing the program recorded in the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.
Further, in the present specification, the term “system” means an overall apparatus configured by a plurality of devices, a plurality of means, and the like.

  As mentioned above, although several embodiment of this invention was described, these embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take other various embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are also included in the invention described in the claims and the equivalents thereof.

The invention described in the scope of claims at the beginning of the filing of the present application will be appended.
[Appendix 1]
Image acquisition means for acquiring data of a plurality of captured images captured by each of a plurality of imaging devices that capture a predetermined space in which an object can move from different positions;
Position acquisition means for acquiring the position of the object moving in the predetermined space;
Based on the position acquired by the position acquisition means, the data of the plurality of captured images acquired by the image acquisition means is processed for viewpoint conversion, so that the virtual viewpoint arranged in the object is a predetermined point. Image processing means for generating image data representing a view of the viewing direction as viewpoint converted image data;
Output control means for executing control for outputting data of the viewpoint conversion image generated by the image processing means;
An image processing apparatus comprising:
[Appendix 2]
The position acquisition means acquires the position of the object moving in the predetermined space at an arbitrary timing,
The image processing means processes the data of the captured image acquired by the image acquisition means based on the position of the object acquired at an arbitrary timing by the position acquisition means.
The image processing apparatus according to appendix 1, wherein:
[Appendix 3]
The position acquisition means sequentially acquires the position of the object moving in the predetermined space,
The image processing means sequentially processes the data of the captured image acquired by the image acquisition means based on the position of the object sequentially acquired by the position acquisition means.
The image processing apparatus according to Supplementary Note 2, wherein
[Appendix 4]
Among the data of the plurality of captured images acquired by the image acquisition unit, original position data that is a source of generation of an image representing a state viewed from a virtual viewpoint arranged on the object is obtained by the position acquisition unit. An original image selection means for selecting based on the position acquired by
The image processing means processes the original image data acquired by the original image selection means to perform viewpoint conversion, thereby representing an image representing a state in which a virtual viewpoint placed on the object looks at a predetermined viewing direction. Is generated as viewpoint-converted image data.
The image processing apparatus according to any one of supplementary notes 1 to 3, wherein
[Appendix 5]
The position acquisition means further acquires the moving direction of the object,
The original image selection means selects the data of the original image based on the moving direction of the object acquired by the position acquisition means,
The image processing means further processes the data of the original image based on the movement direction acquired by the position acquisition means, so that a virtual viewpoint arranged on the object moving in the movement direction becomes the movement direction. Is generated as data of the viewpoint conversion image, the image data representing the state of viewing as the viewing direction,
The image processing apparatus according to appendix 4, wherein
[Appendix 6]
The position acquisition means further acquires the moving speed of the object,
The image processing means generates image data of an angle of view corresponding to the moving speed acquired by the position acquisition means as the viewpoint converted image data.
The image processing apparatus according to appendix 4 or 5, characterized in that:
[Appendix 7]
The image processing means generates data of an image having a wide angle in inverse proportion to the moving speed, as data of the viewpoint conversion image.
The image processing apparatus according to appendix 6, wherein:
[Appendix 8]
The original image selection unit selects data of a plurality of original images from the data of the plurality of captured images captured by the plurality of imaging units,
The image processing means generates data of the viewpoint conversion image based on the data of the plurality of original images selected by the original image selection means.
The image processing apparatus according to any one of appendices 4 to 7, characterized in that:
[Appendix 9]
The object obtains and transmits data indicating its own state,
The position acquisition means acquires the data transmitted from the object, and acquires the state of the object based on the data;
The image processing apparatus according to any one of appendices 1 to 8, wherein
[Appendix 10]
A receiving device that receives the data indicating the state of the object transmitted from the object is provided outside the image processing device,
The position acquisition means acquires the data indicating the state of the object via the receiving device.
The image processing apparatus according to appendix 9, wherein:
[Appendix 11]
A generation source for generating an electromagnetic field and a receiving means for receiving and detecting the electromagnetic field are provided outside the image processing apparatus,
The position acquisition means acquires the state of the object based on a reception detection result of the reception means;
The image processing apparatus according to any one of appendices 1 to 8, wherein
[Appendix 12]
A beacon radio antenna that transmits radio waves of different frequencies is provided outside the image processing device,
The position acquisition means acquires the state of the object using the Doppler effect by the radio wave transmitted from the beacon radio antenna.
The image processing apparatus according to any one of appendices 1 to 8, wherein
[Appendix 13]
The position acquisition means performs image processing on captured image data in which the object is captured among the plurality of captured image data acquired by the image acquisition means, and based on the result of the image processing, Get object state,
The image processing apparatus according to any one of appendices 1 to 8, wherein
[Appendix 14]
Identification information is attached to the object,
The position acquisition means recognizes the identification information by the image processing, and acquires the state of the object using the identification information.
The image processing apparatus according to attachment 13, wherein the image processing apparatus is characterized in that
[Appendix 15]
The position acquisition means recognizes the appearance of the object by the image processing, and acquires the state of the object based on the appearance.
The image processing apparatus according to attachment 13, wherein the image processing apparatus is characterized in that
[Appendix 16]
There are a plurality of objects in the space captured by the plurality of imaging devices,
The position acquisition means acquires the state of each of the plurality of objects,
The image processing means generates data of the viewpoint conversion image based on the state of the selected object among the states of the plurality of objects acquired by the position acquisition means.
The image processing apparatus according to any one of appendices 1 to 15, characterized in that:
[Appendix 17]
The output control means executes control to display and output the data of the viewpoint conversion image generated by the image processing means.
The image processing apparatus according to any one of supplementary notes 1 to 16, wherein:
[Appendix 18]
The output control unit functions as a transmission unit that transmits data of the viewpoint conversion image generated by the image processing unit to an external device.
18. The image processing apparatus according to any one of appendices 1 to 17, wherein
[Appendix 19]
It further comprises control receiving means for receiving a control signal from an external device,
The image processing means generates data of an image representing a state viewed from a virtual viewpoint instructed by a control signal received by the control receiving means as data of the viewpoint conversion image.
The image processing apparatus according to any one of appendices 1 to 18, wherein
[Appendix 20]
An image processing method executed by an image processing apparatus that executes control to output viewpoint-converted image data,
An image acquisition step of acquiring data of a plurality of captured images captured by each of a plurality of imaging devices that capture a predetermined space in which an object can move from different positions;
A position acquisition step of acquiring a position of the object moving in the predetermined space;
Based on the position acquired in the position acquisition step, the virtual viewpoint arranged in the object has a predetermined viewing direction by processing the captured image data acquired in the image acquisition step for viewpoint conversion. An image processing step for generating data of an image representing a state of viewing the image as viewpoint conversion image data;
An output control step for executing control to output data of the viewpoint conversion image generated by the image processing step;
An image processing method including:
[Appendix 21]
To a computer that controls an image processing apparatus that executes control to output data of a viewpoint-converted image,
An image acquisition function for acquiring data of a plurality of captured images captured by each of a plurality of imaging devices that capture a predetermined space in which an object can move from different positions;
A position acquisition function for acquiring the position of the object moving in the predetermined space;
Based on the position acquired by the position acquisition function, the viewpoint of the captured image acquired by the image acquisition function is processed to convert the virtual viewpoint arranged in the object into a predetermined viewing direction. An image processing function for generating image data representing the state of viewing the image as viewpoint converted image data,
An output control function for executing control to output data of the viewpoint conversion image generated by the image processing function;
A program that realizes functions including

  DESCRIPTION OF SYMBOLS 10 ... Image processing system, 11a-11h ... Camera, 12 ... Image processing apparatus, 13 ... Relay apparatus, 14-1 to 14-N ... External terminal, 15-1 to 15- DESCRIPTION OF SYMBOLS 3 ... Receiving device, 16-1 to 16-3 ... Magnetic field generation source, 17-1 to 17-3 ... Receiving means, 31 ... CPU, 32 ... ROM, 33 ... RAM, 34 ... image processing unit, 35 ... bus, 36 ... input / output interface, 37 ... input unit, 38 ... display unit, 39 ... timing unit, 40 ... memory 41, communication unit, 42, drive, 51, removable media, 61 ... main control unit, 62 ... captured image acquisition unit, 63 ... position information acquisition unit, 64. ..Storage control unit, 65... Display control unit, 66... Communication control unit, 71. 72 ... Moving speed calculation unit, 73 ... Original image selection unit, 74 ... Viewpoint conversion image processing unit, 81 ... Original image candidate data storage unit, 82 ... Object data storage unit, 83 ... Display data storage unit, 101 ... Imaging unit, 102 ... Transmission unit, 111 ... Sensor unit, 112 ... Transmission unit, A1, A2 ... Moving direction, F ... Competition Field, G1, G2 ... Goal, M1-M7 ... Mark, N ... Network, OB1-OB7 ... Object

Claims (21)

Image acquisition means for acquiring data of a plurality of captured images captured by each of a plurality of imaging devices that capture a predetermined space in which an object can move from different positions;
Position acquisition means for acquiring the position of the object moving in the predetermined space;
Based on the position acquired by the position acquisition means, the data of the plurality of captured images acquired by the image acquisition means is processed for viewpoint conversion, so that the virtual viewpoint arranged in the object is a predetermined point. Image processing means for generating image data representing a view of the viewing direction as viewpoint converted image data;
Output control means for executing control for outputting data of the viewpoint conversion image generated by the image processing means;
An image processing apparatus comprising: The position acquisition means acquires the position of the object moving in the predetermined space at an arbitrary timing,
The image processing means processes the data of the captured image acquired by the image acquisition means based on the position of the object acquired at an arbitrary timing by the position acquisition means.
The image processing apparatus according to claim 1. The position acquisition means sequentially acquires the position of the object moving in the predetermined space,
The image processing means sequentially processes the data of the captured image acquired by the image acquisition means based on the position of the object sequentially acquired by the position acquisition means.
The image processing apparatus according to claim 2. Among the data of the plurality of captured images acquired by the image acquisition unit, original position data that is a source of generation of an image representing a state viewed from a virtual viewpoint arranged on the object is obtained by the position acquisition unit. An original image selection means for selecting based on the position acquired by
The image processing means processes the original image data acquired by the original image selection means to perform viewpoint conversion, thereby representing an image representing a state in which a virtual viewpoint placed on the object looks at a predetermined viewing direction. Is generated as viewpoint-converted image data.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The position acquisition means further acquires the moving direction of the object,
The original image selection means selects the data of the original image based on the moving direction of the object acquired by the position acquisition means,
The image processing means further processes the data of the original image based on the movement direction acquired by the position acquisition means, so that a virtual viewpoint arranged on the object moving in the movement direction becomes the movement direction. Is generated as data of the viewpoint conversion image, the image data representing the state of viewing as the viewing direction,
The image processing apparatus according to claim 4. The position acquisition means further acquires the moving speed of the object,
The image processing means generates image data of an angle of view corresponding to the moving speed acquired by the position acquisition means as the viewpoint converted image data.
The image processing apparatus according to claim 4, wherein the image processing apparatus is an image processing apparatus. The image processing means generates data of an image having a wide angle in inverse proportion to the moving speed, as data of the viewpoint conversion image.
The image processing apparatus according to claim 6. The original image selection unit selects data of a plurality of original images from the data of the plurality of captured images captured by the plurality of imaging units,
The image processing means generates data of the viewpoint conversion image based on the data of the plurality of original images selected by the original image selection means.
The image processing apparatus according to claim 4, wherein the image processing apparatus is an image processing apparatus. The object obtains and transmits data indicating its own state,
The position acquisition means acquires the data transmitted from the object, and acquires the state of the object based on the data;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A receiving device that receives the data indicating the state of the object transmitted from the object is provided outside the image processing device,
The position acquisition means acquires the data indicating the state of the object via the receiving device.
The image processing apparatus according to claim 9. A generation source for generating an electromagnetic field and a receiving means for receiving and detecting the electromagnetic field are provided outside the image processing apparatus,
The position acquisition means acquires the state of the object based on a reception detection result of the reception means;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A beacon radio antenna that transmits radio waves of different frequencies is provided outside the image processing device,
The position acquisition means acquires the state of the object using the Doppler effect by the radio wave transmitted from the beacon radio antenna.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The position acquisition means performs image processing on captured image data in which the object is captured among the plurality of captured image data acquired by the image acquisition means, and based on the result of the image processing, Get object state,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Identification information is attached to the object,
The position acquisition means recognizes the identification information by the image processing, and acquires the state of the object using the identification information.
The image processing apparatus according to claim 13. The position acquisition means recognizes the appearance of the object by the image processing, and acquires the state of the object based on the appearance.
The image processing apparatus according to claim 13. There are a plurality of objects in the space captured by the plurality of imaging devices,
The position acquisition means acquires the state of each of the plurality of objects,
The image processing means generates data of the viewpoint conversion image based on the state of the selected object among the states of the plurality of objects acquired by the position acquisition means.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The output control means executes control to display and output the data of the viewpoint conversion image generated by the image processing means.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The output control unit functions as a transmission unit that transmits data of the viewpoint conversion image generated by the image processing unit to an external device.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. It further comprises control receiving means for receiving a control signal from an external device,
The image processing means generates data of an image representing a state viewed from a virtual viewpoint instructed by a control signal received by the control receiving means, as the viewpoint converted image data.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. An image processing method executed by an image processing apparatus that executes control to output viewpoint-converted image data,
An image acquisition step of acquiring data of a plurality of captured images captured by each of a plurality of imaging devices that capture a predetermined space in which an object can move from different positions;
A position acquisition step of acquiring a position of the object moving in the predetermined space;
Based on the position acquired in the position acquisition step, the virtual viewpoint arranged in the object has a predetermined viewing direction by processing the captured image data acquired in the image acquisition step for viewpoint conversion. An image processing step for generating data of an image representing a state of viewing the image as viewpoint conversion image data;
An output control step for executing control to output data of the viewpoint conversion image generated by the image processing step;
An image processing method including: To a computer that controls an image processing apparatus that executes control to output data of a viewpoint-converted image,
An image acquisition function for acquiring data of a plurality of captured images captured by each of a plurality of imaging devices that capture a predetermined space in which an object can move from different positions;
A position acquisition function for acquiring the position of the object moving in the predetermined space;
Based on the position acquired by the position acquisition function, the viewpoint of the captured image acquired by the image acquisition function is processed to convert the virtual viewpoint arranged in the object into a predetermined viewing direction. An image processing function for generating image data representing the state of viewing the image as viewpoint converted image data,
An output control function for executing control to output data of the viewpoint conversion image generated by the image processing function;
A program that realizes functions including

Images