JP2022042153A - Three-dimensional shooting model generation system and three-dimensional shooting model generation method - Google Patents

Abstract

To provide a three-dimensional shooting model generation system that can be realized with a simple configuration and that anyone can easily enjoy and handle.SOLUTION: A three-dimensional shooting model generation system includes cameras 201 to 204 which are a plurality of photographing devices 20 for photographing a subject OB from a plurality of directions, and a controller 110 and an image processing server 120 which are arithmetic processing devices 10 configured to generate a three-dimensional photographing model of the subject OB on the basis of image data obtained by photographing the subject OB by the plurality of photographing devices 201 to 204. The depth of field DOF of an image pickup lens system 221 used in the cameras 201 to 204 is set to ±0.5 to ±1.5 m centered on a focal position CIF on an optical axis Ax.SELECTED DRAWING: Figure 2

Description

The present invention relates to a live-action stereoscopic photography technique (Volumetric Capture) for generating a three-dimensional model of a subject based on image data of the subject taken from a plurality of directions.

In recent years, a live-action stereoscopic photography technique (Volumetric Capture) that can generate a three-dimensional model of a subject by photographing the movement of a target person or object from all directions of 360 degrees without using a marker has attracted attention. There is. According to this technology, by extracting a subject for each frame from moving images of people taken synchronously by multiple cameras and creating a three-dimensional model, the subject model can be placed at any position in the virtual space and its movement can be measured. It can be freely reproduced on the display. An image obtained by arithmetically processing an image taken from a plurality of viewpoints and reproduced so as to be visible from an arbitrary viewpoint is called a "free viewpoint image" or a "virtual viewpoint image". For example, in Patent Document 1, such a virtual image is used. A system for generating a viewpoint image is disclosed.

Japanese Unexamined Patent Publication No. 2017-21827

In the Volumetric Capture, it is necessary to remove the background from the frame image taken as described above and extract the target subject. Conventionally, as a method for distinguishing between a subject and a background, a method of projecting mesh light on the subject and a method of using a green background as the background have been adopted. However, with these conventional methods, it is necessary to install a projector, a green background, or the like for projecting mesh light in the shooting place, and shooting can only be performed in a studio equipped with equipment. It is also conceivable to consider a method of extracting a subject by performing an object extraction process on the captured image data. However, in order to extract objects (subject images) one by one for each frame of high-definition moving image data of several tens of frames per second taken from multiple viewpoints, a large platform with high-speed and large-capacity processing capacity is required. It becomes.

The present invention has been made in view of such conventional problems, and is a stereoscopic imaging model generation system that can be realized with a simple configuration and can be easily enjoyed and handled by anyone, and a stereoscopic image executed by the system. The purpose is to provide a method for generating a shooting model.

In order to solve the above-mentioned problems, the present invention has a plurality of photographing devices for photographing a subject from a plurality of directions, and the subject based on image data obtained by the plurality of photographing devices photographing the subject. A stereoscopic photography model generation system including an arithmetic processing device configured to generate a stereoscopic photography model of the above, in which the depth of field of the image pickup lens system used in the image pickup device is in focus on the optical axis. It is a stereoscopic photography model generation system set to ± 0.5 to ± 1.5 m around the center.

The stereoscopic photography model generation system has an edge detection accuracy when the subject is outside the depth of field, whereas the maximum value of the edge detection accuracy of the contour when the subject is within the depth of field. The ratio of is preferably 0.7 or less.

Further, in the stereoscopic imaging model generation system, it is preferable that the edge detection accuracy is calculated based on the value obtained by first-derivating the change in luminance at the boundary of the contour. Alternatively, the stereoscopic imaging model generation system may be one in which the edge detection accuracy is calculated based on a value obtained by quadrically differentiating the change in luminance at the boundary of the contour.

Further, in the stereoscopic photography model generation system, it is preferable that the arithmetic processing unit has a free viewpoint image generation unit that generates a free viewpoint image based on the stereoscopic photography model.

The present invention also includes a plurality of photographing devices and an arithmetic processing device that generates a stereoscopic photographing model of the subject based on image data obtained by photographing the subject by the plurality of photographing devices. This is a stereoscopic photography model generation method performed in a system in which the depth of field of the image pickup lens system used in the above is set to ± 0.5 to ± 1.5 m centered on the position of focus on the optical axis. A step of photographing the subject from a plurality of directions by the plurality of photographing devices, and a step of detecting the contour of the subject from the image data photographed by the plurality of photographing devices in the process executed by the arithmetic processing device. It is a three-dimensional image generation method including a step of generating a stereoscopic photography model of the subject based on the subject image data imaged by the detected contour.

In the three-dimensional image generation method, the edge detection accuracy when the subject is outside the depth of field is compared with the maximum value of the edge detection accuracy of the contour when the subject is within the depth of field. The ratio of is preferably 0.7 or less.

Further, in the three-dimensional image generation method, it is preferable that the edge detection accuracy is calculated based on the value obtained by first-derivating the change in luminance at the boundary of the contour. Alternatively, the three-dimensional image generation method may be one in which the edge detection accuracy is calculated based on a value obtained by quadrically differentiating the change in luminance at the boundary of the contour.

Further, in the three-dimensional image generation method, it is preferable that the process executed by the arithmetic processing unit further includes a step of generating a free viewpoint image based on the stereoscopic imaging model.

Further, in the three-dimensional image generation method, the image data obtained by photographing the subject by the plurality of photographing devices is the moving image data, and the arithmetic processing apparatus synchronizes the plurality of the moving image data with the subject. It is preferable to generate a dynamic 3D imaging model of.

According to the present invention, it is possible to provide a stereoscopic photography model generation system that can be realized with a simple configuration and that anyone can easily enjoy and handle, and a stereoscopic photography model generation method executed by the system.

It is a block diagram of the stereoscopic photography model generation system by one Embodiment of this invention. It is a figure for demonstrating the positional relationship between a camera and a depth of field. It is a figure for demonstrating the method of measuring edge detection accuracy (sharpness) using a test pattern. It is a functional block diagram of a camera adapter. It is a functional block diagram of a control box and its peripheral part. It is a functional block diagram of an image processing server. It is a block diagram of the stereoscopic photography model generation system by another embodiment of this invention.

The stereoscopic photography model generation system according to the preferred embodiment of the present invention includes an arithmetic processing unit that performs processing such as generating a three-dimensional model of the subject, and a plurality of imaging devices for photographing the subject from a plurality of directions. It is composed of. In this specification, "subject" and "object" taken by a camera are used interchangeably. The subject is not limited to a person but may be an object (movable object). Further, the "three-dimensional shooting model" of the object generated by this system is referred to as a "three-dimensional model" or a "three-dimensional moving image model". Further, in the embodiment described below, the "image" is assumed to be a moving image, and the "three-dimensional model" includes a time parameter for expressing the motion, but in carrying out the present invention, the "image" is used. It should be clearly stated that still images such as photographs are included.

Hereinafter, a specific embodiment will be described with respect to an example in which an object (subject) OB is used as a person and the movement of the person on the spot is modeled three-dimensionally for a certain period of time.

FIG. 1 is a block diagram of a stereoscopic imaging model generation system according to an embodiment of the present invention. The arithmetic processing unit 10 includes a controller 110 on the user side and an image processing server 120 which is a cloud server system on the Internet N. The control box 111, which is the main body of the controller 110, can be a dedicated computer device or a general-purpose PC (Personal Computer). Further, the controller 110 may include a terminal device 113 such as a smartphone or a tablet PC wirelessly connected to the control box 111 in order to assist the operation by the user.

The photographing apparatus 20 includes a plurality of cameras 201 to 204 capable of capturing digital moving images. The main bodies of the cameras 201 to 204 are equipped with camera adapters 211 to 214, respectively. In the embodiment of FIG. 1, the photographing apparatus 20 is composed of four cameras, but the number of cameras is not limited to this, and may be appropriately increased or decreased in the range of, for example, about 3 to 24 cameras.

The cameras 201 to 204 are installed on a tripod (not shown) placed at predetermined positions on four corners and / or sides of, for example, a 5 m square mat 230 laid on the floor. The image pickup lens system 221 of each camera 201 to 204 is directed in advance to the center of the mat 230 on which the subject OB stands. The mat 230 is preferably a green mat in order to distinguish it from the subject and facilitate the extraction process of the subject. Further, the shape of the mat is not limited to a quadrangle, and may be a polygon or a circle depending on the number of cameras.

As shown in FIG. 2, the image pickup lens system 221 used in the cameras 201 to 204 of the present embodiment has a central position where the depth of field (DOF) of the image pickup lens system 221 is exactly the focus of the lens on the optical axis Ax. It is set to ± 0.5 to ± 1.5 m centering on (CIF; Center in Focus). However, it is assumed that the optical characteristics of the cameras 201 to 204 are common, and there is no variation in the depth of field characteristics themselves between the cameras 201 to 204.

The depth of field refers to the range (field) of the distance on the subject side in which the image of the subject appears to be in focus. Images taken with a camera in a distant view or a near view rather than the depth of field are blurred. A "deep" depth of field means a relatively wide range of focus, and a "shallow" depth of field means a relatively narrow range of focus. As will be described in detail later, the stereoscopic photography model generation system of the present embodiment is characterized in that it uses a lens having a shallower depth of field than the conventional one.

The depth of field characteristics of a camera are also affected by the focal length and aperture (F value) of the image pickup lens. As shown in FIG. 2, the image pickup lens system 221 according to the present embodiment is centered on the position CIF where the image is exactly focused according to the shooting environment and conditions such as the size of the shooting location and the range in which the object (person) OB moves. A lens having a characteristic of a depth of field of ± 0.5 to ± 1.5 m is selectively adopted. In the embodiment of FIG. 2, the distance from the camera 201 to the position CIF at the center of the depth of field was 3.5 m, and the F value was 1.

There is a permissible circle of confusion diameter as a criterion for determining whether or not a certain position on the optical axis is within the depth of field, in other words, as a criterion for measuring the boundary of the depth of field on the optical axis. However, as in the present embodiment, the depth of field or the boundary of the field may be determined based on the edge detection accuracy (also referred to as “sharpness”) of the subject contour. More specifically, the ratio of the edge detection accuracy when the subject is outside the depth of field is 0.7 or less to the maximum value of the edge detection accuracy when the subject is within the depth of field. The boundary can be determined by whether or not it is (that is, 30% less than the maximum value).

As shown in FIG. 3, for example, the edge detection accuracy (sharpness) of the contour can be accurately measured based on the brightness change of the boundary portion where the black and white are inverted by using the test pattern TP in which the black and white are clearly separated. can. First, the maximum value SEmax of sharpness when the test pattern TP is at the central position CIF of the depth of field DOF is measured. Next, the sharpness SE is measured while gradually changing the position of the test pattern TP on the optical axis Ax to the side closer to the camera 201 (minus direction) and the side farther from the camera 201 (plus direction) around the CIF. Then, both the front and rear positions where the sharpness ratio SE / SEmax is 0.7 can be determined as the boundary of the depth of field DOF.

The edge detection accuracy can be calculated, for example, based on the value obtained by first-derivating the change in luminance at the boundary of the contour (gradient method). In this gradient method, the sharpness can be calculated by the size of the vertical axis of the maximum value or the minimum value obtained by firstly differentiating the luminance. Further, the edge detection accuracy may be calculated based on a value obtained by quadratic differentiation of the change in luminance at the boundary of the contour (Laplacian method). In this Laplacian method, sharpness can be calculated by the magnitude of the amplitude on the horizontal axis that constitutes the inflection of luminance. The method for calculating the edge detection accuracy is not limited to the gradient method and the Laplacian method described above.

Next, the camera adapters 211 to 214 are connected to the switching hub 300 via a communication cable 240 conforming to a standard such as Ethernet (registered trademark) or USB (Universal Serial Bus). The controller 110 (control box 111) acquires image data taken by each camera 201 to 204 and camera parameter information via the switching hub 300.

The system of the embodiment of FIG. 1 has a so-called star-shaped network structure in which the cameras 201 to 204 and the camera adapters 211 to 214 are connected to one switching hub 300. However, in order to support higher definition such as 4K or 8K of captured images and larger capacity due to higher frame rate, multiple cameras 201 to 204 are connected by a daisy chain in which adjacent camera adapters are connected in cascade. May be good.

FIG. 4 is a functional block diagram inside the camera adapter 211. Since the other camera adapters 212 to 214 have the same configuration as this, their description will be omitted. As shown in the figure, the camera adapter 211 is roughly divided into a camera control unit 215, a correction processing unit 216, and a frame generation / transmission unit 217. The camera control unit 215 has a lens adjustment unit 2151, a shooting operation control unit 2152, and a shooting parameter setting unit 2153.

The lens adjustment unit 2151 electrically adjusts parameters such as focus, zoom, and exposure of the image pickup lens system 221 of the camera 201. The focus and the like of the image pickup lens system 221 may be adjusted automatically, or may be manually performed by a command from an external device (for example, the controller 110) or by the user while looking at the monitor screen.

The shooting operation control unit 2152 controls the start and stop operations of shooting in synchronization with a Genlock signal or the like from an external device (for example, the controller 110).

The shooting parameter setting unit 2153 sets shooting parameters such as image resolution (number of shooting pixels), color depth, frame rate, and white balance, and submits the shooting operation control unit 2152. The shooting parameter setting may be specified from the outside (for example, the controller 110), or the shooting parameter setting unit 2153 may automatically set the shooting parameter.

The correction processing unit 216 performs color correction processing, blur correction processing due to camera vibration, lens-specific distortion correction processing, and the like on the raw image data held in the cache memory 218.

The frame generation / transmission unit 217 formats and converts the image data on the frame memory 219 to generate the frame data. Meta information (attached information) such as time code indicating the shooting time, identifier information of the shooting camera, camera parameters (focus value, zoom value, lens characteristics, etc.) is added to the frame data and filed. .. The communication adapter 2171 of the frame generation / transmission unit 217 transmits the created frame image data to the controller 110.

Next, the configuration of the controller 110 will be described. FIG. 5 is a functional block diagram of the control box 111, which is the main body of the controller 110, and its peripheral portion. The control box 111 has a data input unit 1111, a data synchronization unit 1112, a calibration unit 1113, a data compression unit 1114, a storage 1115, a network adapter unit 1116, a format conversion unit 1117, a rendering processing unit 1118, and a user interface unit 1119. ing.

The data input unit 1111 is connected to the camera adapters 211 to 214 via the communication cable 240 and the switching hub 300, and receives the frame image data which is the data of the images taken by the cameras 201 to 204 in real time.

The data synchronization unit 1112 temporarily stores the acquired image data in an internal high-speed DRAM (Dynamic Random Access Memory), and buffers the data until all the data of the cameras 201 to 204 are prepared. A time code (time information) and camera identifier information are assigned to the image data, and the data synchronization unit 1112 performs time synchronization control of each image data based on each time code. Further, the data synchronization unit 1112 may synchronize each image data by using, for example, a feature point matching method.

Due to a slight difference in distance between each camera 201 to 204 and the subject OB, a difference in characteristics peculiar to the lens, and the like, the dimensions of the subject OB imprinted on each image with respect to the absolute coordinates may differ in each image. The calibration unit 1113 performs a process of normalizing the size of each image in order to calibrate such a dimensional error. Further, the calibration unit 1113 may simultaneously perform correction to match the brightness and contrast of each image data.

The synchronized and normalized image data is compressed into a predetermined format such as MPEG by the data compression unit 1114, and transmitted to the image processing server 120 via the network adapter unit 1116. As will be described later, the image processing server 120 is a platform on the cloud that performs high-speed arithmetic processing for generating a three-dimensional moving image model of a subject (person) OB based on frame image data of cameras 201 to 204. Further, the image data may be stored in an internal storage 1115, which is, for example, a large-capacity SSD (Solid State Drive).

The 3D moving image model data of the subject OB sent back from the image processing server 120 to the network adapter unit 1116 is processed by the format conversion unit 1117 such as data expansion, and the time information and the coordinate information are separated to form a frame. It is converted to each polygon mesh model data. Then, the converted polygon mesh model data is stored in the storage 1115 in association with time information such as a frame rate.

The rendering processing unit 1118 converts the three-dimensional model of the subject OB stored in the storage 1115 into a two-dimensional free-viewpoint image that can be reproduced and displayed on the display 112 or an external monitoring device (for example, a smartphone 113). The user can instruct the rendering processing unit 1118 to indicate the viewpoint position by operating the smartphone 113, if necessary, via the user interface unit 119. The viewpoint may be at any position around 360 ° of the subject OB, and the height of the viewpoint can be arbitrarily changed within a limited range. In addition, the user can give an instruction to arbitrarily move the viewpoint position to change the direction of the three-dimensional model moving image being reproduced. Further, the user can also display an arbitrary viewpoint image in which a specific portion of the three-dimensional model image is zoomed in or out on the display 112 or the like.

Next, the configuration of the image processing server 120 will be described with reference to the functional block diagram of FIG. The image processing server 120 mainly has an input / output unit 1201, an image processing unit 1202, and a routing processing unit 1203.

The input / output unit 1201 includes a data receiving unit 1204 and a data transmitting unit 1205. The image data of the subject OB taken from a plurality of directions by the plurality of cameras 201 to 204 is received by the data receiving unit 1204 of the input / output unit 1201 via the control box 111 and the Internet N.

The image processing unit 1202 includes a preprocessing unit 1206, an object extraction unit 1207, and a three-dimensional model generation unit 1208. The preprocessing unit 1206 separates meta information such as camera identifier information, camera parameters, and control box 111 address information from the received image data, and performs predetermined decompression processing to extract frame data from the object extraction unit 1207. Send to.

At this stage, the frame image includes the subject OB and the background around it. The object extraction unit 1207 performs a process of extracting an image area in which the subject OB is captured by detecting the edge of the contour included in the image data of each frame. As the edge detection method, it is preferable to use a gradient method based on the value obtained by first-derivating the change in luminance at the boundary of the contour, or a Laplacian method based on the value obtained by making the change in luminance second derivative.

In the present embodiment, as described above, the depth of field of the lenses of the cameras 201 to 204 is set to be relatively shallow, ± 0.5 to ± 1.5 m. Therefore, in the captured image, only the area of the subject OB is in focus, and the surrounding background area is blurred. Therefore, without installing a green background in the background or projecting mesh light onto the subject OB, only the edge of the contour of the subject OB is detected, and the image is relatively easily and accurately drawn by the contour of the subject OB. The image area (subject image data) to be created can be extracted.

The subject image data separated from the background image is sent to the three-dimensional model generation unit 1208. The 3D model generation unit 1208 generates a 3D moving image model of the subject OB based on the subject image data taken from a plurality of directions, for example, using the principle of a stereo camera.

The routing processing unit 1203 sets the address of the control box 111 to which the original image data is transmitted as the transfer destination of the generated three-dimensional moving image model data. Then, the data transmission unit 1205 transmits the generated three-dimensional moving image model data to the control box 111 of the set address on the Internet N.

Although one preferred embodiment of the present invention has been described above, various modifications can be made in carrying out the present invention without departing from the spirit of the present invention. For example, the image processing server 120 on the Internet N may have an image data processing function executed by the control box 111 described above. Further, the stand-alone type control box 111 may execute a series of arithmetic processes for generating a three-dimensional model without using a cloud server.

Further, as shown in FIG. 7, the photographing device 20 may be configured by a portable electronic device having a small camera, for example, smartphones 251 to 254. Dedicated lens units 261 to 264 for setting a shallow depth of field of the camera are attached to the camera openings of the smartphones 251 to 254. Application software for the user to receive the stereoscopic photography model generation service is installed in the smartphones 251 to 254 for photography and the smartphones 255 for operation control. The user can use the application of the smartphone 255 to transmit the image data of the subject OB taken by the plurality of smartphones 251 to 254 to the arithmetic processing device 10 via the Internet N by the switching hub 300 and the line modem 310.

The stereoscopic image of the subject OB and / or its free viewpoint image generated and processed by the arithmetic processing unit 10 can be transferred or downloaded to the smartphone 255 via the Internet N. As a result, the user can operate the smartphone 255 to easily reproduce the stereoscopic image of the subject OB on the screen.

According to the stereoscopic photography model generation system of the embodiment described above and the stereoscopic photography model generation method executed by the system, only the subject is focused by using an image pickup lens having a relatively shallow depth of field for the image pickup device. , It is possible to take a moving image with a blurred background around it. In addition, if such a moving image with a shallow depth of field is used, the subject image can be simply edged without installing a green background at the shooting location or a projector for projecting mesh light onto the subject. It can be easily and accurately separated and extracted from the background image by the detection process. Therefore, the image data processing load until the subject is extracted can be significantly reduced, so that even a general user can easily handle the image, and it is possible to provide a system having a simple configuration.

10 Arithmetic processing device 20 Imaging device 110 Controller 111 Control box 112 Display 113 Terminal device (smartphone)
120 Image processing server 201-204 Camera 211-214 Camera adapter 220 Image sensor (CCD)
221 Imaging lens system 230 Matt 240 Communication cable 251 to 255 Smartphone 261 to 264 Lens unit 300 Switching hub 1110 Synchronization signal generation unit 1112 Data synchronization unit 1115 Storage 1116 Network adapter unit 1118 Rendering processing unit 1119 User interface unit 1201 Input / output unit 1202 Image Processing unit 1203 Routing processing unit 1207 Object extraction unit 1208 Three-dimensional model generation unit N Internet Ax Optical axis CIF Focused position DOF Depth of field OB Subject, object SE Edge detection accuracy, sharpness TP test pattern

The present invention relates to a live-action stereoscopic photography technique (Volumetric Capture) for generating a three-dimensional model of a subject based on image data of the subject taken from a plurality of directions.

In recent years, a live-action stereoscopic photography technique (Volumetric Capture) that can generate a three-dimensional model of a subject by photographing the movement of a target person or object from all directions of 360 degrees without using a marker has attracted attention. There is. According to this technology, by extracting a subject for each frame from moving images of people taken synchronously by multiple cameras and creating a three-dimensional model, the subject model can be placed at any position in the virtual space and its movement can be measured. It can be freely reproduced on the display. An image obtained by arithmetically processing an image taken from a plurality of viewpoints and reproduced so as to be visible from an arbitrary viewpoint is called a "free viewpoint image" or a "virtual viewpoint image". For example, in Patent Document 1, such a virtual image is used. A system for generating a viewpoint image is disclosed.

In the Volumetric Capture, it is necessary to remove the background from the frame image taken as described above and extract the target subject. Conventionally, as a method for distinguishing between a subject and a background, a method of projecting mesh light on the subject and a method of using a green background as the background have been adopted. However, in these conventional methods, it is necessary to install a projector, a green background, or the like for projecting mesh light in the shooting place, and shooting can only be performed in a studio equipped with equipment. It is also conceivable to consider a method of extracting a subject by performing an object extraction process on the captured image data. However, in order to extract objects (subject images) one by one for each frame of high-definition moving image data of several tens of frames per second taken from multiple viewpoints, a large platform with high-speed and large-capacity processing capacity is required. It becomes.

In order to solve the above-mentioned problems, the present invention has a plurality of photographing devices for photographing a subject from a plurality of directions, and the subject based on image data obtained by the plurality of photographing devices photographing the subject. A stereoscopic photography model generation system including an arithmetic processing device configured to detect the contour of the subject and generate a stereoscopic photography model of the subject, wherein the depth of field of the image pickup lens system used in the image pickup device is determined. This is a stereoscopic photography model generation system set to ± 0.5 to ± 1.5 m around the focal position on the optical axis.

The present invention also includes a plurality of photographing devices and an arithmetic processing device that generates a stereoscopic photography model of the subject based on image data obtained by the plurality of photographing devices shooting the subject. This is a stereoscopic photography model generation method performed in a system in which the depth of field of the image pickup lens system used in the above is set to ± 0.5 to ± 1.5 m centered on the position of focus on the optical axis. A step of photographing the subject from a plurality of directions by the plurality of photographing devices, and a step of detecting the contour of the subject from the image data photographed by the plurality of photographing devices in the process executed by the arithmetic processing device. It is a stereoscopic photography model generation method including a step of generating a stereoscopic photography model of the subject based on the subject image data imaged by the detected contour.

In the stereoscopic photography model generation method, the edge detection accuracy when the subject is outside the depth of field is compared with the maximum value of the edge detection accuracy of the contour when the subject is within the depth of field. The ratio of is preferably 0.7 or less.

Further, in the stereoscopic photography model generation method, it is preferable that the edge detection accuracy is calculated based on the value obtained by first-derivating the change in luminance at the boundary of the contour. Alternatively, the three-dimensional image generation method may be one in which the edge detection accuracy is calculated based on a value obtained by quadrically differentiating the change in luminance at the boundary of the contour.

Further, in the stereoscopic photography model generation method, it is preferable that the process executed by the arithmetic processing unit further includes a step of generating a free viewpoint image based on the stereoscopic photography model.

Further, in the stereoscopic photography model generation method, the image data obtained by photographing the subject by the plurality of photographing devices is the moving image data, and the arithmetic processing unit synchronizes the plurality of the moving image data with the subject. It is preferable to generate a dynamic stereoscopic photography model of.

As shown in FIG. 2, the image pickup lens system 221 used in the cameras 201 to 204 of the present embodiment has a central position where the depth of field (DOF) of the image pickup lens system 221 is exactly the focus of the lens on the optical axis Ax. It is set to ± 0.5 to ± 1.5 m centering on (CIF; Center in Focus). However, it is assumed that the optical characteristics of the cameras 201 to 204 are common, and there is no variation in the depth of field characteristics themselves between the cameras 201 to 204.

The frame generation / transmission unit 217 formats and converts the image data on the frame memory 219 to generate the frame data. Meta information (attached information) such as time code indicating the shooting time, identifier information of the shooting camera, camera parameters (focus value, zoom value, lens characteristics, etc.) is added to the frame data and filed. .. The frame generation / transmission unit 217 transmits the created frame image data to the controller 110.

Claims (11)

Multiple shooting devices for shooting subjects from multiple directions, and
A stereoscopic imaging model generation system including an arithmetic processing unit configured to generate a stereoscopic imaging model of the subject based on image data obtained by photographing the subject by the plurality of imaging devices.
A stereoscopic photography model generation system in which the depth of field of the image pickup lens system used in the photographing apparatus is set to ± 0.5 to ± 1.5 m centered on the position of focus on the optical axis. The ratio of the edge detection accuracy when the subject is outside the depth of field is 0.7 or less to the maximum value of the edge detection accuracy of the contour when the subject is within the depth of field. The stereoscopic photography model generation system according to claim 1. The stereoscopic photography model generation system according to claim 2, wherein the edge detection accuracy is calculated based on a value obtained by first-derivating a change in luminance at the boundary of the contour. The stereoscopic photography model generation system according to claim 2, wherein the edge detection accuracy is calculated based on a value obtained by secondarily differentiating the change in luminance at the boundary of the contour. The stereoscopic photography model generation system according to any one of claims 1 to 4, wherein the arithmetic processing apparatus has a free viewpoint image generation unit that generates a free viewpoint image based on the stereoscopic photography model. An image pickup lens system used in the image pickup device, which includes a plurality of image pickup devices and an arithmetic processing device that generates a stereoscopic photography model of the subject based on image data obtained by the plurality of image pickup devices taking a picture of the subject. This is a stereoscopic photography model generation method performed in a system in which the depth of field of is set to ± 0.5 to ± 1.5 m centered on the position of focus on the optical axis.
A step of photographing the subject from a plurality of directions by the plurality of photographing devices, and a step of photographing the subject from a plurality of directions.
The processing executed by the arithmetic processing unit is
A step of detecting the contour of the subject from image data taken by the plurality of photographing devices, and a step of detecting the contour of the subject.
A three-dimensional image generation method including a step of generating a stereoscopic shooting model of the subject based on the subject image data drawn by the detected contour. The ratio of the edge detection accuracy when the subject is outside the depth of field is 0.7 or less to the maximum value of the edge detection accuracy of the contour when the subject is within the depth of field. The three-dimensional image generation method according to claim 6. The stereoscopic photography model generation method according to claim 7, wherein the edge detection accuracy is calculated based on a value obtained by first-derivating the change in luminance at the boundary of the contour. The stereoscopic photography model generation method according to claim 7, wherein the edge detection accuracy is calculated based on a value obtained by secondarily differentiating the change in luminance at the boundary of the contour. The stereoscopic imaging model generation method according to any one of claims 6 to 9, wherein the processing executed by the arithmetic processing unit further includes a step of generating a free viewpoint image based on the stereoscopic imaging model. The image data obtained by photographing the subject by the plurality of photographing devices is the moving image data.
The stereoscopic imaging model generation method according to any one of claims 6 to 10, wherein the arithmetic processing unit synchronizes a plurality of the moving image data to generate a dynamic stereoscopic imaging model of the subject.

Images