WO2024070618A1

WO2024070618A1 - Imaging device, program, and imaging system

Info

Publication number: WO2024070618A1
Application number: PCT/JP2023/032982
Authority: WO
Inventors: フォレストマシュー; 恭男湯山; 英二新谷; 桐郎増井; 知嗣南川
Original assignee: ソニーグループ株式会社
Priority date: 2022-09-26
Filing date: 2023-09-11
Publication date: 2024-04-04

Abstract

This imaging device comprises: an imaging unit which outputs a captured image; a display control unit which causes a display unit to display a synthesized image generated by synthesizing a CG image on the captured image on the basis of spatial information about an imaging space and CG parameters pertaining to CG synthesis; and an association unit which associates the captured image and the CG parameters.

Description

Imaging device, program, imaging system

This technology relates to imaging devices, programs, and imaging systems, and in particular to technology for synthesizing captured images with CG (Computer Graphics) images.

A system has been proposed that can synthesize virtual space images created using 3D CG with real space images captured by a camera, and display the images in real time.

JP 2011-35638 A

Incidentally, there is a demand for further improvements in operational efficiency in video production, including CG compositing.

The aim of this technology is to improve work efficiency in video production, including CG compositing.

An imaging device according to the present technology includes an imaging unit that outputs an imaged image, a display control unit that causes a display unit to display a composite image generated by combining a CG image with the imaged image based on spatial information of an imaging space and CG parameters related to CG synthesis, and an association unit that associates the imaged image with the CG parameters.
This allows the imaging device to display a composite image, in which the captured image is combined with the CG image, on the display device in almost real time while the imaging device is capturing an image. In addition, since the imaging device associates the captured image with the CG parameters, it is possible to generate a CG image based on the CG parameters even later.

FIG. 1 is a diagram illustrating an imaging system according to an embodiment. 2A to 2C are diagrams illustrating a captured image, a CG image, and a composite image. FIG. 2 is a diagram illustrating the external configuration of the imaging device. FIG. 2 is a diagram illustrating a functional configuration of the imaging apparatus. FIG. 2 is a diagram illustrating a configuration of a computer. FIG. 2 is a diagram illustrating a processing process of the imaging system. FIG. 13 is a diagram showing a sequence chart of a preparation process. FIG. 4 is a diagram illustrating distance information. FIG. 1 is a diagram illustrating a process for generating a CG file. FIG. 2 is a diagram illustrating an example of CG parameters. FIG. 2 is a diagram illustrating a CG image based on a CG file and CG parameters. FIG. 2 is a diagram illustrating meta information. FIG. 11 is a diagram showing a sequence chart of an imaging process. FIG. 2 is a diagram illustrating imaging settings of an imaging device. FIG. 13 is a diagram illustrating a composite image. FIG. 13 is a diagram illustrating an adjustment screen. FIG. 13 is a diagram illustrating an adjustment screen. FIG. 13 is a diagram illustrating an adjustment screen. 11A to 11C are diagrams illustrating screens displayed on a display unit in the main imaging process. FIG. 13 is a diagram illustrating an editing screen. 11 is a diagram illustrating the functional configuration of a camera control unit and a computer control unit in Modification 1. FIG. FIG. 11 is a block diagram illustrating an imaging process in the first modified example. 11 is a diagram illustrating the functional configuration of a camera control unit and a computer control unit in Modification 2. FIG. FIG. 11 is a block diagram illustrating an imaging process in Modification 2.

The embodiments will be described below in the following order.
1. Overview of the imaging system
2. Imaging device
<3. Configuration of Computer 3>
4. Processing steps of imaging system 1
5. Modifications
<6. Summary>

In addition, in this technology, "imaging" includes not only imaging that involves recording of image data, but also imaging for displaying an image on a display unit without recording of image data, such as a so-called through image or live view image.
The term "image" does not only refer to an image displayed on a display unit, but may also refer to image data that is not displayed on a display unit.

1. Overview of the imaging system
Fig. 1 is a diagram illustrating an imaging system 1 according to an embodiment. Fig. 2A is a diagram illustrating a captured image 101. Fig. 2B is a diagram illustrating a CG image 102. Fig. 2C is a diagram illustrating a composite image 103. As shown in Fig. 1, the imaging system 1 includes an imaging device 2 and a computer 3.

The imaging device 2 is placed, for example, at an imaging site, and captures an image of a subject at the imaging site to generate an image (moving image) 101 as shown in FIG. 2A. There may be only one imaging device 2, but here we will explain a case where two imaging devices 2 are provided. When explaining the two imaging devices 2 separately, one imaging device 2 will be referred to as imaging device 2a and the other imaging device 2 will be referred to as imaging device 2b.

The imaging devices 2 are connected to each other, for example, wirelessly, and can transmit and receive images and various information to and from each other. Note that the imaging devices 2 may be connected to each other by wires, or may be connected to each other via a cloud server or the like.
In FIG. 1, the imaging device 2 a is held by a cameraman 8 and captures an image of a subject in response to the operation of the cameraman 8 .
In FIG. 1, the imaging device 2b is fixed at a predetermined position and in a predetermined direction by, for example, a tripod, and captures an image of a subject in synchronization with the imaging device 2a using a known synchronization method.
The

imaging devices

2a and 2b are capable of capturing images of a subject from different positions and directions. In the following, the

imaging devices

2a and 2b will be described as capturing images without moving from a predetermined position, but they may be configured to capture images while moving.

The subject can be anything. Here, as an example of the subject, we will explain a case where three people 6 are running from the back (farther from the imaging device 2) to the front (closer to the imaging device 2) on a sidewalk 5 adjacent to a river 4, and there is a bridge 7 across the river 4 and sidewalk 5.

The computer 3 is assumed to be placed at a location different from the imaging site, but may be placed at the imaging site. The computer 3 is connected to the imaging device 2, for example, wirelessly, and receives the captured image 101 from the imaging device 2 and transmits the composite image 103 to the imaging device 2.

When the computer 3 receives an captured image 101 from the imaging device 2 (one or both of the

imaging devices

2a and 2b), if it is necessary to synthesize a CG image 102 with the captured image 101, the computer 3 generates a CG image 102 as shown in FIG. 2B based on spatial information and meta information described later.
Then, the computer 3 composites the captured image 101 with the CG image 102 to generate a composite image 103 as shown in Fig. 2C. Note that the composite image 103 may also include frames in which the CG image 102 is not composited. After that, the computer 3 transmits the generated composite image 103 to, for example, the imaging device 2a.

When the imaging device 2a receives the composite image 103 from the computer 3, it displays the received composite image 103 on the display unit 13 (see FIG. 3). This makes it possible for the imaging system 1 to allow the cameraman 8 to check the composite image 103 as a moving image in almost real time while the imaging device 2 is capturing an image. Details of the imaging system 1 will be described below.

2. Imaging device
Fig. 3 is a diagram illustrating the external configuration of the imaging device 2. Fig. 4 is a diagram illustrating the functional configuration of the imaging device 2.
As shown in FIG. 3, the imaging device 2 includes an imaging unit 11, a ToF (Time of Flight) sensor 12, a display unit 13, and an operation unit .

As shown in FIG. 4, the imaging unit 11 includes, for example, a lens system 21, an imaging element unit 22, a camera processing unit 23, a recording unit 24, a communication unit 25, a camera control unit 26, a memory unit 27, and a driver unit 28.

The lens system 21 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, etc. The lens system 21 collects light (incident light) from a subject onto the image sensor unit 22.
The lens system 21 may be provided integrally with the imaging device 2 , or may be configured as an interchangeable lens separate from the imaging device 2 .

The imaging element unit 22 is configured to have an image sensor (imaging element), such as a complementary metal oxide semiconductor (CMOS) type or a charge coupled device (CCD) type.
In the image sensor unit 22, light received by the image sensor is photoelectrically converted to obtain an electric signal, and the electric signal is subjected to, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, and A/D (Analog/Digital) conversion processing. The image sensor unit 22 can output image data as digital data after A/D conversion processing to the camera process unit 23, the recording unit 24, and the communication unit 25 in the subsequent stages.

The camera processor 23 is configured as an image processor using, for example, a DSP (Digital Signal Processor).
The camera processor 23 generates image data in a predetermined format by performing various signal processes on the digital data (image signal) from the image sensor 22. For example, the camera processor 23 performs lens correction, noise reduction, synchronization processing, YC generation processing, color reproduction/sharpness processing, file formation processing, etc.

In the synchronization process, a color separation process is performed so that the image data for each pixel has all color components of R, G, and B. For example, in the case of an image sensor using a mosaic color filter such as a Bayer array, a demosaic process is performed as the color separation process.
In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from R, G, and B image data.
In the color reproduction/sharpness processing, processing is carried out to adjust gradation, saturation, tone, contrast, etc., as the so-called image creation.
In the file creation process, image data is subjected to, for example, compression encoding for recording or communication, formatting, generation and addition of meta information, etc., to generate files for recording or communication.
For example, it is conceivable to generate image files in the MP4 format, which is used for compressing moving images conforming to MPEG-4, or in the XAVC format.

The recording unit 24 is a memory card (such as a portable flash memory) which is a recording medium that can be attached to and detached from the imaging device 2, or a flash memory or HDD (Hard Disk Drive) built into the imaging device 2. The recording unit 24 records image data output from the imaging element unit 22 or image data output from the camera process unit 23. The recording unit 24 also records meta information for generating a CG image 102 to be combined with the captured image 101 in association with the image data. The meta information will be described in detail later.
The image data output from the imaging element unit 22 is RAW image data that has not been subjected to image processing by the camera process unit 23. Therefore, the recording unit 24 may record RAW image data, or may record image data that has been subjected to image processing by the camera process unit 23.
In the following, it is assumed that RAW image data is recorded in the recording unit 24. Note that the RAW image data may be either uncompressed RAW image data or compressed RAW image data.

The display unit 13 is composed of a display device such as a liquid crystal display (LCD) panel or an organic electroluminescence (EL) display, and the number of such display units 13 may be any number. In the example of Fig. 3, two display units 13 are provided, but the display area of one display unit 13 may be divided into two, and the two divided display areas may be used as display areas for images from two imaging devices.
The display unit 13 may be provided integrally with the imaging unit 11 or may be provided separately. As an example of the display unit 13 being provided separately from the imaging unit 11, for example, a display unit of a smartphone may be used as the display unit 13. In this case, it is sufficient that the imaging device 2 and the smartphone are connected wirelessly or by wire and are capable of transmitting and receiving images and information to each other. Then, the display control unit 33 described later transmits the image via the communication unit 25, and the communication unit 25 receives a control signal in response to an operation from the smartphone.
The display unit 13 displays various images on a display screen. For example, the display unit 13 displays a reproduced image of image data read from the recording unit 24.
The display unit 13 also displays various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface) on the screen.

The communication unit 25 communicates wirelessly or wired with external devices (other imaging devices 2, computer 3). For example, the communication unit 25 transmits the captured image 101 to the computer 3 and receives the composite image 103.

The operation unit 14 collectively refers to input devices for the user to input various operations. Specifically, the operation unit 14 includes various operators (keys, dials, touch panel) provided in the imaging unit 11, a touch panel 14a provided on the front surface of the display unit 13, and the like.
When the operation unit 14 detects a user operation, a signal corresponding to the input operation is sent to the camera control unit 26 .

The camera control unit 26 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The camera control unit 26 includes functional units as a spatial information acquisition unit 31, a meta information acquisition unit 32, a display control unit 33, a recording control unit 34, and an adjustment unit 35. Details of these functional units will be described later.
The memory unit 27 stores information and the like used for processing by the camera control unit 26. The illustrated memory unit 27 collectively represents, for example, a read only memory (ROM), a random access memory (RAM), a flash memory, and the like.
The memory unit 27 may be a memory area built into the microcomputer chip serving as the camera control unit 26, or may be configured by a separate memory chip.
The camera control unit 26 executes programs stored in the ROM of the memory unit 27, the recording unit 24, and the like, thereby controlling the entire imaging device 2.
For example, the camera control unit 26 controls the operation of each necessary unit with respect to controlling the shutter speed of the image sensor unit 22, instructing various signal processing in the camera processing unit 23, image capturing and recording operations in response to user operations, playback operations of recorded image data, operations of the lens system 21 such as zoom, focus, and aperture adjustment, user interface operations, etc.

The RAM in the memory unit 27 is used as a working area for various data processing by the CPU of the camera control unit 26, and is used for temporarily storing data, programs, and the like.
The ROM and flash memory (non-volatile memory) in the memory unit 27 are used to store the OS (Operating System) that the CPU uses to control each unit, application programs for various operations, firmware, various setting information, etc.

The driver unit 28 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, and the like.
These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 26, thereby moving the focus lens and zoom lens, opening and closing the aperture blades of the aperture mechanism, and so on.

The ToF sensor 12 includes, for example, an irradiation unit that irradiates infrared laser light (irradiation light), and a sensor, for example of a CMOS type, that receives the light that is irradiated by the irradiation unit and reflected by the subject. The ToF sensor 12 calculates the distance from the imaging device 2 to the subject using the ToF (Time of Flight) method.

The ToF sensor 12 can use a direct ToF method that calculates distance by directly determining the time it takes for irradiated light to return as reflected light after being reflected by the subject, or an indirect ToF method that calculates distance to the subject based on the phase difference between irradiated light and reflected light.

<3. Configuration of Computer 3>
FIG. 5 is a diagram illustrating the configuration of the computer 3.
The computer 3 is assumed to be a personal computer (PC), a mobile terminal device such as a smartphone or a tablet, a mobile phone, a video editing device, a video playback device, etc. The computer 3 may also be configured as a server device or a computing device in cloud computing.
Therefore, the computer 3 may be a smartphone used as the display unit 13 , or may be an imaging device 2 .
Furthermore, the computer 3 may be a different device for each process or for one or more functional units, which will be described later.

The computer 3 includes a computer control unit 41, a recording unit 42, a display unit 43, a communication unit 44, an operation unit 45, and a memory unit 46.

The computer control unit 41 is composed of a microcomputer (arithmetic processing unit) equipped with a CPU. The computer control unit 41 has functional units as a CG correction unit 51, an image acquisition unit 52, a CG image generation unit 53, a CG synthesis unit 54, and an editing unit 55. Details of these functional units will be described later.

The recording unit 42 is a memory card (such as a portable flash memory) which is a recording medium that can be attached to and detached from the computer 3, or a flash memory or HDD (hard disk drive) built into the computer 3, or the like.
The recording unit 42 records the captured image 101 (RAW image data) received from the imaging device 2, and records a CG file or the like that includes various information for generating a CG image 102 to be combined with the captured image 101. Note that the captured image 101 received from the imaging device 2 may be image data that has been subjected to various signal processes by the camera processing unit 23.

The display unit 43 is a display unit that displays various information to the user, and is configured to include, for example, a liquid crystal panel or an organic EL display.

The communication unit 44 communicates with an external device (imaging device 2) wirelessly or via a wired connection. For example, the communication unit 44 receives the captured image 101 from the imaging device 2 and transmits the composite image 103.

The operation unit 45 collectively represents input devices for the user to input various operations, specifically, the operation unit 45 includes various operators (keyboard, mouse, keys, dial, touch panel, etc.).
When the operation unit 45 detects a user operation, a signal corresponding to the input operation is sent to the computer control unit 41 .

The memory unit 46 collectively indicates, for example, a ROM, a RAM, a flash memory, etc. The memory unit 46 may be a memory area built into the microcomputer chip serving as the computer control unit 41, or may be configured by a separate memory chip.
The computer control unit 41 controls the entire computer 3 by executing programs stored in the ROM of the memory unit 46, the recording unit 42, etc.

The RAM in the memory unit 46 is used as a working area for various data processing by the CPU of the computer control unit 41, for temporarily storing data, programs, etc.
The ROM and flash memory in the memory unit 46 are used to store the OS for the CPU to control each unit, application programs for various operations, various setting information, and the like.

4. Processing steps of imaging system 1
Fig. 6 is a diagram for explaining the processing steps of the imaging system 1. Here, the steps performed by the imaging system 1 will be explained. As shown in Fig. 6, the processing steps of the imaging system 1 are roughly divided into three stages: a preparation step ST1, an imaging step ST2, and a post-processing step ST3.

The preparation process ST1 is a process carried out prior to the imaging process ST2. In the preparation process ST1, spatial information (spatial map) of the imaging space is acquired by the imaging device 2, and the acquired spatial information is used by the computer 3 to determine and modify CG parameters for generating the CG image 102.

The imaging process ST2 is a process in which imaging is actually performed at an imaging site such as that shown in FIG. 1. This imaging process ST2 includes a rehearsal imaging process in which rehearsal imaging is performed prior to actual imaging, and a main imaging process in which actual imaging is performed. In the rehearsal imaging process and the main imaging process, a CG image 102 is synthesized in almost real time with an imaging image 101 obtained by the imaging device 2, and is displayed on the display unit 13 as a synthesized image 103. In addition, in the rehearsal imaging process, a process of adjusting CG parameters is performed in response to the operation of the operation unit 14.

The post-processing step ST3 indicates various processes that are carried out after shooting. For example, CG parameter adjustment, image adjustment, clip editing, video effects, etc. are carried out.

Image adjustments may include color adjustments, brightness adjustments, contrast adjustments, and the like.
Clip editing may involve cutting clips, adjusting the order, adjusting the length of time, and the like.
As an image effect, special effect image synthesis may be performed.

These steps are described in detail below.

[4.1. Preparation process]
Fig. 7 is a diagram showing a sequence chart of the preparation process Fig. 8 is a diagram for explaining distance information.

In the preparation process ST1, the ToF sensor 12 of one of the imaging devices 2 (e.g., imaging device 2a) is used to sense the imaging space that will be the background in the actual imaging. The cameraman 8 may sense the imaging space while moving the imaging device 2a so that the ToF sensor 12 can sense the distance to the subject in the imaging space that will be captured as the background in the actual imaging. At this time, it is preferable that there are no moving subjects, such as the person 6 shown in Figure 1.

In the imaging device 2a, the imaging unit 11 may acquire an image 101 in addition to acquiring distance information in the ToF sensor 12. In this case, it is preferable to match the imaging frame rate and imaging timing of the imaging unit 11 with the sensing rate and sensing timing of the ToF sensor 12.
However, it is not essential that the imaging frame rate and imaging timing of the imaging unit 11 do not match the sensing rate and sensing timing of the ToF sensor 12, i.e., that they are synchronized, and distance information corresponding to the imaging timing of each frame of the captured image may be calculated by interpolation processing or the like.

The ToF sensor 12 detects the distance to the subject, for example, according to the sensing rate. The spatial information acquisition unit 31 of the imaging device 2a acquires distance information indicating the distance to the subject from the ToF sensor 12 (step S1). Then, the spatial information acquisition unit 31 acquires spatial information (spatial map) of the imaging space from the acquired distance information (step S2). Note that in FIG. 8, the imaging space for which spatial information has been acquired is shown by a mesh. Also, areas without a mesh are areas for which spatial information has not yet been acquired.
The spatial information may include not only the position (coordinates) of the imaging space relative to a predetermined reference position (here, the position of the imaging device 2a), but also the shape of structures in the imaging space and their relative positions relative to the reference position when the imaging device 2 captures images.
The reference position may be, for example, the position where the imaging device 2 is placed during actual imaging, or may be any other position.

Here, the spatial information acquisition unit 31 may calculate spatial information using the captured image 101 acquired by the imaging unit 11 in addition to or instead of the distance information acquired from the ToF sensor 12. For example, the distance information may be calculated by image processing from the captured image 101 acquired by the imaging unit 11, and further, continuous areas of the same color may be determined to be one structure based on color information of the captured image 101. This makes it possible to calculate the spatial information with high accuracy.
Although the spatial information will be described as three-dimensional spatial information, it may be two-dimensional spatial information (depth map). Furthermore, by acquiring the captured image 101 together, it is possible to obtain spatial information including texture.

The communication unit 25 transmits the acquired spatial information to the computer 3 (step S3). The computer 3 receives the spatial information transmitted from the imaging device 2a (step S11). This enables the computer 3 to modify the CG parameters based on the received spatial information and CG file.

FIG. 9 is a diagram explaining the CG parameter correction process. FIG. 10 is a diagram explaining an example of CG parameters.

In the computer 3, a CG correction screen 62 as shown in FIG. 9 is displayed on the display unit 43. The user can determine and modify desired CG parameters by operating the CG correction screen 62 via the operation unit 45. Note that the CG correction screen 62 is merely one example, and any configuration is acceptable as long as it is possible to determine and modify CG parameters using spatial information.

The CG correction unit 51 displays a CG image 102 based on the three-dimensional imaging space indicated in the spatial information transmitted from the imaging device 2 in the CG display area 63 of the CG correction screen 62.

The CG correction unit 51 regenerates the CG image 102 while appropriately applying corrections based on user operations on the CG correction screen 62, displays it in the CG display area 63, and corrects the CG parameters in accordance with the user operations (step S12).

In the CG correction screen 62, it is possible to determine and correct CG parameters such as the placement area, type, color, size, number (density), target subject, start timing, drawing speed, shading (light source direction, intensity, color temperature) of CG objects such as plants and stones. These CG parameters can be said to be parameters for generating the CG image 102.

The CG modification unit 51 determines and modifies a placement area 63a in which a CG object is placed in the three-dimensional imaging space shown in the spatial information in response to a user operation on the CG display area 63. The placement area 63a determined and modified here is indicated, for example, by an address (composite position) on the CG image 102 in which the CG object is drawn. However, the placement area 63a may also be an address in the imaging space in which the CG object is placed.

In addition to a CG display area 63, the CG correction screen 62 displays a type selection area 64 for selecting the type of CG object, a number change dial 65 for changing the number of CG objects, a color wheel 66 for selecting the hue of the CG image 102, and a seek bar 67 for changing the drawing speed of the CG image 102.

The CG modification unit 51 determines the CG object selected by the user in response to operation of the operation unit 45 from the multiple CG object types displayed in the type selection area 64 as the type of CG object to be rendered as the CG image 102.
Furthermore, the CG correction unit 51 corrects the number of CG objects to be drawn as CG images 102 in the placement area 63a to a number (density) corresponding to the value selected by the number-changing dial 65. Note that the number-changing dial 65 is rotated and displayed in response to a user operation via the operation unit 45. Therefore, the CG correction unit 51 variably determines the number of CG objects to be drawn as CG images 102 in the placement area 63a in response to the operation of the rotated and displayed number-changing dial 65.
Furthermore, the CG correction section 51 corrects the CG image 102 to a color tone selected by a color wheel 66 as the color tone for rendering the CG image 102 .

The start timing of the CG parameters here indicates the timing at which the composition of the CG image 102 starts with the captured image 101, i.e., the start trigger. The drawing speed indicates the time from the start to the end of composition of the CG image 102. Note that the drawing speed may be specified by the number of frames rather than by time.

The CG correction unit 51 determines and corrects the target subject and start timing in response to the user's operation of the operation unit 45. Specifically, the CG correction unit 51 determines the target subject in response to the user's operation, and can specify the distance to a specific subject as the start timing, and the synthesis start position indicating the position at which synthesis will start when the target subject is located, using an address in the imaging space.

The CG correction unit 51 can also change the drawing speed of the CG image 102 in response to the operation of the seek bar 67. For example, moving the seek bar 67 to the left increases the time from when the synthesis of the CG image 102 begins to when it ends. Moving the seek bar 67 to the right decreases the time from when the synthesis of the CG image 102 begins to when it ends.

10, suppose that a specific person 6 is specified as the target subject, the spatial address of the imaging space is specified as (x, y, z) = (10, 50, 30) as the start timing, two seconds is specified as the drawing speed, (X, Y) = (10, 50) is specified as the placement area (composite position), and three CG objects are specified as the number of objects.
In this case, synthesis of the CG image 102 starts when the specific person 6 reaches (x, y, z) = (10, 50, 30) in the imaging space. At this time, the CG image 102 is generated in which a CG object is drawn at (X, Y) = (10, 50). Note that the specific person 6 may be identified by image processing of the captured image 101. The position (address) of the specific person 6 in the imaging space is calculated based on distance information (position information in space) obtained from the ToF sensor 12.
Also, a CG image 102 is generated for each frame in which, for example, three CG objects gradually appear forward at predetermined intervals over a period of two seconds from the start of composition.

The CG modification unit 51 associates the modified CG parameters and spatial information with the CG file and records them in the recording unit 42 (step S13).

FIG. 11 is a diagram illustrating a CG image 102 based on CG parameters. As described above, the CG parameters are modified in response to user operations. As a result, the CG image generation unit 53 generates the CG image 102 based on the modified CG parameters.

As shown in FIG. 11 from top to bottom, it is therefore possible to generate a CG image 102 based on CG parameters in which, for example, plants and rocks appear gradually from the rear to the front.

Fig. 12 is a diagram for explaining meta information. As shown in Fig. 12, the meta information includes a CG file ID, CG parameters, and imaging setting information.
As described above, the CG file ID is an identifier for identifying the determined CG file.
The in-space position information is information indicating the distance to the subject sensed by the ToF sensor 12, that is, the position (address) of the subject in the imaging space.
The imaging setting information is information regarding imaging settings of the imaging device 2 that are set in an imaging step ST2 described later.
The meta information includes the start timing, which is necessary to determine when to display the CG image 102, but as long as the CG image 102 can be composited at the appropriate timing and position, it may also include information other than the CG file ID, CG parameters, distance information, and imaging setting information, or it may not include any of this information.

[4.2. Imaging process]
Next, the imaging process will be described. The imaging process is mainly divided into a rehearsal imaging process and a main imaging process. In the rehearsal imaging process, the imaging device 2 is caused to image the subject in the same manner as in the main imaging process. Note that the rehearsal imaging process does not necessarily have to be provided.

FIG. 13 is a diagram showing a sequence chart in the imaging process. FIGS. 14A to 14D are diagrams explaining the imaging settings of the imaging device 2. When the rehearsal imaging process is started, an imaging setting screen (camera setup screen) 71 for determining the imaging settings of the imaging device 2 is displayed on the display unit 13 of the imaging device 2a. Note that the display order of the screens shown in FIGS. 14A to 14D does not have to be the order described below, and may be any other order.

As shown in FIG. 14A, the imaging setting screen 71 displays operation items for selecting and setting the resolution (Imager Scan) of the image sensor of the imaging element section 22, the number of images vertically and horizontally, i.e., the image size, the video format (Video Format) indicating either interlaced or progressive, the frame rate of the output image (Project FPS), the 3D LUT (Monitor LUT) for color correction of the so-called live view image (monitor image) that is displayed on the display section 13 for monitoring during imaging, and the color space (Color Space) (step S21).

When each operation item is operated via the touch panel 14a, a list of options that can be selected for the operated selection item is displayed. For example, when the operation item for frame rate (Project FPS) is operated, a selection screen 72 is displayed that lists the selectable frame rates, as shown in FIG. 14B. When any value on the selection screen 72 is selected via the touch panel 14a, the frame rate is changed to the selected value, and the imaging setting screen 71 shown in FIG. 14A is displayed again.

When the imaging settings for all operation items have been determined, the meta information acquisition unit 32 acquires imaging setting information indicating the determined imaging settings as meta information, and the recording control unit 34 records them in the recording unit 24.

Furthermore, once the imaging settings have been determined, a CG selection screen 73 for selecting a CG file is displayed, as shown in FIG. 14C. Here, the names and IDs of the selectable CG files are displayed. Then, when any CG file is selected on the CG selection screen 73 via operation of the touch panel 14a, the selected CG file is confirmed.

Once the CG file is determined, the meta information acquisition unit 32 acquires the CG file ID of the determined CG file as meta information, and the recording control unit 34 records it in the recording unit 24.

After that, as shown in FIG. 14D, an imaging device selection screen 74 for selecting the imaging device 2 to be used in the rehearsal imaging process and the main imaging process is displayed on the display unit 13. Then, when the imaging device 2 to be used in the rehearsal imaging process and the main imaging process is determined by operating the touch panel 14a, the communication unit 25 transmits imaging setting information to the determined imaging device 2 directly or via the computer 3. This makes it possible to cause multiple imaging devices 2 to perform imaging with the same imaging settings. Note that here, for example, an imaging device 2 that can be linked to a smartphone as the display unit 13 is selected.

The communication unit 25 also transmits the imaging setting information, CG file ID, and imaging device 2 of the determined imaging device 2 to the computer 3 as meta information (step S21). The computer 3 records the received imaging setting information and CG file ID of the imaging device 2 in the recording unit 42 as meta information.

When the various settings in the rehearsal imaging process are thus completed, the imaging device 2 starts rehearsal imaging in response to an operation by the cameraman 8 or the like (step S22). Note that, when there are two imaging devices 2, the two imaging devices 2 start rehearsal imaging in synchronization with each other.
When the rehearsal imaging is started, the imaging device 2 obtains an image 101. The camera processor 23 sequentially performs image processing on the image 101 (RAW image data). The communication unit 25 sequentially transmits the image 101 (image data) after image processing to the computer 3. Note that the communication unit 25 may transmit the RAW image data to the computer 3.

In addition, the meta information acquisition unit 32 acquires distance information that indicates the distance to the subject calculated by the ToF sensor 12, and the communication unit 25 transmits the distance information to the computer 3 as spatial position information.

15 is a diagram illustrating a composite image 103. The computer 3 first receives meta information from the captured image 101. After that, the computer 3 receives the spatial position information almost in real time. The CG image generating unit 53 then reads out the CG file and CG parameters corresponding to the CG file ID included in the meta information from the recording unit 42, and uses the CG file to generate a CG image 102 to be composited with each frame of the captured image 101 based on the CG parameters and the spatial position information (step S31).
Thereafter, the CG synthesis unit 54 synthesizes each frame of the captured image 101 received from the imaging device 2 with the CG image 102 generated by the CG image generation unit 53 to generate a composite image 103. Note that the captured image 101 may be recorded in the recording unit 42.

Then, the communication unit 44 transmits the generated composite image 103 to the imaging device 2a (step S32).
Here, the composite image 103 is transmitted only to the image capturing device 2a that has been preset, while the composite image 103 is not transmitted to the image capturing device 2b that has not been preset.

When the imaging device 2a receives the composite image 103, the display control unit 33 displays the received composite image 103 on the display unit 13 (step S23).

In this way, in the imaging device 2a, as shown from the top to the bottom of Figure 15, it is possible to display a composite image 103 on the display unit 13 almost in real time while the subject is being imaged.

In other words, in the example of FIG. 15, a composite image 103 in which plants, stones, etc. appear in time with the running of a specific person 6 is displayed on the display unit 13.

FIG. 16 is a diagram illustrating the adjustment screen. In response to user operations during the rehearsal imaging process, the display unit 13 of the imaging device 2a displays a composite image display area 81 in which the composite image 103 is displayed, a number change dial 82 for changing the number (density) of CG objects, and a color wheel 83 for changing the color tone of the CG image 102, as shown in FIG. 16. The CG parameters can be adjusted by operating the number change dial 82 and color wheel 83. At this time, the imaging device 2a obtains the CG parameters from the computer 3 and records them in the recording unit 24.

When the number-changing dial 82 is operated, the adjustment unit 35 changes (adjusts) the number (density) of CG objects in response to the operation. Note that the number-changing dial 82 is displayed and rotated in response to a user operation via the touch panel 14a.
17, when the number-changing dial 82 is rotated counterclockwise via the touch panel 14a, the number (density) of CG objects is decreased, whereas when the number-changing dial 82 is rotated clockwise via the touch panel 14a, the number (density) of CG objects is increased.

Specifically, when the number-changing dial 82 is operated, the recording control unit 34 updates the number of CG objects among the CG parameters in accordance with the operation of the number-changing dial 82, and records the updated CG parameters in the recording unit 24. In addition, the communication unit 25 transmits the updated CG parameters to the computer 3 (step S24).
When the computer 3 receives the CG parameters, it regenerates the CG image 102 and the composite image 103 based on the received CG parameters, and transmits the composite image 103 to the imaging device 2 (step S33). Note that, in this case, the composite image 103 may be regenerated using the captured image recorded in the recording unit 42.
As a result, the imaging device 2 can display on the display unit 13 a composite image 103 that is composed of a CG image 102 in which the number of CG objects has been reduced in response to operation of the number-changing dial 82, as shown in the lower part of Figure 17, for example.

Furthermore, when the color wheel 83 is operated, the adjustment unit 35 can display a composite image 103 in which the CG image 102 is adjusted to the hue selected by the color wheel 83 .
Specifically, when the color wheel 83 is operated, the recording control unit 34 updates the color tone, which is one of the CG parameters, in accordance with the operation of the color wheel 83 and records the updated color tone in the recording unit 24 .
Furthermore, the communication unit 25 transmits the updated CG parameters to the computer 3. Upon receiving the CG parameters, the computer 3 regenerates the CG image 102 and the composite image 103 based on the received CG parameters, and transmits the composite image 103 to the imaging device 2. This enables the imaging device 2 to display on the display unit 13 the composite image 103 obtained by combining the CG image 102 with a color tone corresponding to the operation of the color wheel 83.

Furthermore, the adjustment unit 35 can display a composite image 103 in which the drawing speed of the CG image 102 is updated by swiping the composite image display area 81 left or right.
18 , for example, by swiping the composite image display area 81 to the left, a composite image 103 in which the drawing speed of the CG image 102 is slowed down is displayed on the display unit 13. By swiping the composite image display area 81 to the right, a composite image 103 in which the drawing speed of the CG image 102 is increased is displayed on the display unit 13.

Specifically, when the composite image display area 81 is swiped left or right, the recording control unit 34 updates the drawing speed, which is one of the CG parameters, in accordance with the amount of the swipe and records the updated CG parameters in the recording unit 24. In addition, the communication unit 25 transmits the updated CG parameters to the computer 3.
When the computer 3 receives the CG parameters, it regenerates the CG image 102 and the composite image 103 based on the received CG parameters, and transmits the composite image 103 to the imaging device 2. As a result, the imaging device 2 can generate a composite image 103 in which the drawing speed is slowed down in accordance with the amount of swiping leftward on the composite image display area 81, and the timing of compositing the front tree among the trees that are CG objects is delayed, as shown in the lower part of Fig. 18, for example.
That is, a composite image 103 can be generated in which the time (number of frames) from the composition start timing to the composition end timing is extended.

In this way, the adjustment unit 35 allows the cameraman 8 or the like to check the composite image 103 displayed in the rehearsal imaging process and allows the cameraman 8 or the like to adjust the CG parameters.

When the CG parameters are adjusted (updated) in the rehearsal imaging process, the adjustment unit 35 records the adjusted CG parameters in the recording unit 24 as meta information, and the communication unit 25 transmits the CG parameters to the computer 3. When the computer 3 receives the adjusted CG parameters, it records the received CG parameters in the recording unit 42 as meta information.

This allows the CG parameters determined and modified in the preparation process ST1 to be adjusted to match the actual captured image 101.

Note that, here, the CG parameters are adjusted using the captured image 101 obtained by the imaging device 2a, but the CG parameters may also be adjusted using the captured image 101 obtained by the imaging device 2b in addition to the imaging device 2a.

In this case, for example, depending on the operation of the touch panel 14a by the cameraman 8 or the like, it may be possible to select whether to generate and display the composite image 103 from the captured image 101 obtained by the imaging device 2a or from the captured image 101 obtained by the imaging device 2b.
Then, a captured image 101 obtained by the

imaging device

2 a or 2 b selected by the cameraman 8 or the like is transmitted to the computer 3 , and the computer 3 generates a CG image 102 and a composite image 103 .
Then, the generated composite image 103 is transmitted to the imaging device 2a, whereby the imaging device 2a displays on the display unit 13 a composite image 103 based on the captured image 101 obtained by the imaging device 2a or the imaging device 2b selected by the cameraman 8 or the like.
This allows the composite image 103 based on the captured image 101 obtained by the imaging device 2b to be confirmed by the imaging device 2a, and the CG parameters can be adjusted by the imaging device 2a using the composite image 103 based on the captured image 101 obtained by the imaging device 2b.

When the rehearsal imaging step is completed, the process moves to the actual imaging step. Fig. 19 is a diagram for explaining a screen displayed on the display unit 13 in the actual imaging step.
In this imaging process, imaging is started in the imaging device 2 as shown in FIG. 1, and a captured image 101 (RAW image data) is obtained (step S25).

The imaging device 2 sequentially transmits the captured images 101 to the computer 3.

Also, similar to the rehearsal imaging process, the meta information acquisition unit 32 acquires distance information indicating the distance to the subject calculated by the ToF sensor 12 as spatial position information and records it as a CG parameter. Also, the communication unit 25 transmits the spatial position information to the computer 3.

The CG image generation unit 53 generates a CG image 102 to be composited with each frame of the captured image 101 based on the selected CG file and meta information. The CG composition unit 54 then composites the CG image 102 with each frame of the received captured image 101 to generate a composite image 103 (step S34).

Once the composite image 103 is generated, the communication unit 44 transmits the composite image 103 to the imaging device 2a (step S35).

By doing this, in the imaging device 2a, it becomes possible to display the composite image 103 on the display unit 13 almost in real time while the subject is being imaged, similar to the rehearsal imaging process (step S26).

At this time, the recording control unit 34 records the captured image 101 in the recording unit 24, and associates the CG file ID, CG parameters, imaging setting information of the imaging device 2, etc., with the image data as meta information. Note that these CG parameters can also be considered parameters for performing post-processing steps.

Here, the term "associate" means, for example, that when processing one piece of information (data, command, program, etc.), the other piece of information can be used (linked). That is, pieces of information associated with each other may be collected into a single file or the like, or may be individual pieces of information. For example, information B associated with information A may be transmitted on a transmission path different from that of information A. Also, for example, information B associated with information A may be recorded on a recording medium different from that of information A (or on a different recording area of the same recording medium). Note that this "association" may be a part of information, not the whole information. For example, an image and information corresponding to the image may be associated with each other in any unit, such as multiple frames, one frame, or a part of a frame.
More specifically, "associating" includes, for example, acts such as assigning the same ID (identification information) to multiple pieces of information, recording multiple pieces of information on the same recording medium, storing multiple pieces of information in the same folder, storing multiple pieces of information in the same file (assigning one to the other as metadata), embedding multiple pieces of information in the same stream, linking multiple pieces of information to the same project, and embedding metadata in an image like a digital watermark.
Specifically, the recording control unit 34 associates the captured image 101 with meta information including the CG parameters and records them in the recording unit 24 .

The communication unit 25 also transmits the captured image 101 and meta information recorded in the recording unit 24 to the computer 3. The computer 3 associates the received captured image 101 and meta information and records them in the recording unit 42. Note that the recording of the captured image 101 and meta information may be performed by only one of the imaging device 2 and the computer 3. This enables a post-processing process.

On the other hand, in both the imaging device 2 and the computer 3, the composite image 103 is discarded without being recorded. Note that the imaging device 2 and the computer 3 may each be able to select a mode in which the composite image 103 is discarded and a mode in which the composite image 103 is recorded without being discarded.

In addition, when there are two imaging devices 2, imaging is started synchronously in the two imaging devices 2. Therefore, when there are two imaging devices 2, the captured images 101 are transmitted to the computer 3 from both the imaging device 2a and the imaging device 2b.
In the computer 3 , the CG image generating section 53 generates a composite image 103 to be composited with each captured image 101 , and the CG composition section 54 composites each captured image 101 with each composite image 103 to generate two composite images 103 .
Here, the composite image 103 may be generated independently for the captured images 101 obtained by the two imaging devices 2, for example, by determining the timing of synthesis based on distance information (space position information) acquired by the ToF sensor 12 of each imaging device 2. Also, the composite image 103 may be generated so as to be related to the captured images 101 obtained by the two imaging devices 2, for example, by determining the timing of simultaneous synthesis based on distance information (space position information) acquired by the ToF sensor 12 of one of the imaging devices 2.

Furthermore, when receiving the composite images 103 from the imaging devices 2, the display control unit 33 may display two of the received composite images 103 side by side on the display unit 13, as shown in FIG.
This allows the cameraman 8 or the like to simultaneously check in almost real time the composite image 103 in which the CG image 102 is composited with the captured images 101 obtained by the

image capture devices

2a and 2b.

[4.3. Post-treatment process]
Next, the post-processing step will be described. The post-processing step is mainly performed by the computer 3.
Fig. 20 is a diagram for explaining an editing screen 91. In the computer 3, an editing screen 91 as shown in Fig. 20 is displayed on the display unit 43 in the post-processing step.

The editing screen 91 includes a composite image display area 92 in which multiple, for example two, composite images 103 can be displayed side-by-side, an adjustment operation area 93 in which multiple icons for adjusting CG parameters are displayed, and a timeline display area 94 in which a timeline of a moving image is displayed.

If the captured image 101 recorded in the recording unit 42 is RAW image data, the editing unit 55 performs image processing on the RAW image data in the same manner as the camera processing unit 23. Here, image processing is performed on each of the captured images 101 obtained by the two imaging devices 2.

In addition, based on the CG files and meta information recorded in the recording unit 42, the editing unit 55 generates CG images 102 to be composited with each captured image 101 in the same manner as the CG image generation unit 53 generated the CG image 102 in this imaging process. Then, the editing unit 55 composites the captured image 101 and the CG image 102 to generate a composite image 103 in the same manner as the CG composition unit 54 generated the composite image 103 in this imaging process. Therefore, here, two composite images 103 are generated.

The editing unit 55 displays the two generated composite images 103 side by side in the composite image display area 92 of the editing screen 91.

The composite image 103 displayed in the composite image display area 92 can be stopped, fast-forwarded, rewound, etc. by operating the operation icons provided at the bottom of the composite image display area 92.
This allows the user to confirm a composite image 103 in which the CG image 102 is composited with the captured image 101 obtained by the imaging device 2. In addition, the user can simultaneously confirm a composite image 103 in which the CG image 102 is composited with the captured images 101 obtained by two imaging devices 2.

Furthermore, the editing unit 55 displays the two captured images 101 and the CG image 102 recorded in the recording unit 42, for example, in the timeline display area 94, one above the other, with their time axes aligned.
This allows the two captured images 101 and the CG image 102 to be simultaneously checked and then edited.

The editing unit 55 can also update the CG parameters in response to operations on icons displayed in the adjustment operation area 93. When the CG parameters are updated, the editing unit 55 records the updated CG parameters in association with the captured image 101, regenerates the CG image 102 based on the updated CG parameters, and generates a composite image 103 by combining the captured image 101 and the CG image 102, and displays it in the composite image display area 92.

At this time, since the CG parameters are adjusted while checking the composite image 103 in the rehearsal shooting step, the adjustment of the CG parameters is reduced in the post-processing step.

5. Modifications
It should be noted that the embodiment is not limited to the specific example described above, and various modified configurations may be adopted.

For example, in the above embodiment, a ToF sensor 12 is provided as a distance measuring unit that measures the distance to the subject, but the distance measuring unit that measures the distance to the subject is not limited to this, and may be an ultrasonic sensor or the like. In addition, detection may be performed by image processing from the output of an image sensor, or multiple distance measuring sensors and image sensors may be used in combination.

In addition, in the above embodiment, the CG file and the CG parameters are recorded separately. However, the CG parameters may be recorded within the CG file.

In addition, in the above embodiment, the camera control unit 26 of the imaging device 2 functions as a spatial information acquisition unit 31, a meta information acquisition unit 32, a display control unit 33, a recording control unit 34, and an adjustment unit 35. However, some or all of these functional units may be made to function in the computer control unit 41 of the computer 3.

The computer control unit 41 also functions as a CG correction unit 51, an image acquisition unit 52, a CG image generation unit 53, a CG synthesis unit 54, and an editing unit 55. However, some or all of these functional units may be controlled by the camera control unit 26 of the imaging device 2.

[5.1. Modification 1]
Fig. 21 is a diagram for explaining the functional configuration of a camera control unit 26A and a computer control unit 41A in the modified example 1. Fig. 22 is a block diagram illustrating an imaging process in the modified example 1.

The camera control unit 26A in the first modification includes a spatial information acquisition unit 31, a meta information acquisition unit 32, a display control unit 33, a recording control unit 34, and an adjustment unit 35, as well as a functional unit serving as a CG synthesis unit 54A. The computer control unit 41 in the first modification also includes functional units serving as a CG correction unit 51, an image acquisition unit 52, a CG image generation unit 53, and an editing unit 55.

In Modification 1, as shown in FIG. 22, a CG image 102 is generated in a computer 3, but a composite image 103 is not generated, and the CG image 102 is transmitted to an imaging device 2. Then, when the imaging device 2 receives the CG image 102, it composites the captured image 101 with the CG image 102 to generate and display the composite image 103.

For example, in the rehearsal process, imaging is started by the imaging element unit 22 (indicated as "imaging" in the figure), and various signal processes are performed by the camera processing unit 23 on the image signal from the imaging element unit 22 (indicated as "camera process" in the figure). Then, the communication unit 25 transmits the captured image 101 that has been subjected to various signal processes to the computer 3.

The CG image generating unit 53 of the computer 3 reads out from the recording unit 42 the CG file corresponding to the CG file ID included in the meta information, and uses the CG file to generate a CG image 102 based on the CG parameters and spatial information (indicated as "CG generation" in the figure). The communication unit 44 then transmits the generated CG image 102 to the imaging device 2a.

When the imaging device 2a receives the CG image 102, the CG synthesis unit 54A synthesizes the CG image 102 with the captured image 101 to generate a synthetic image 103 (indicated as "CG synthesis" in the figure). The display control unit 33 displays the generated synthetic image 103 on the display unit 13 (indicated as "display" in the figure).

Furthermore, when the CG parameters are adjusted in the imaging device 2a (indicated as "CG parameter adjustment" in the figure), the communication unit 25 transmits the adjusted CG parameters to the computer 3. In the computer 3, the CG image generation unit 53 regenerates the CG image 102 based on the received CG parameters, and the communication unit 44 transmits the regenerated CG image 102 to the imaging device 2a.

When the imaging device 2a receives the CG image 102, the CG synthesis unit 54A synthesizes the captured image 101 with the CG image 102 to regenerate a synthetic image 103, and the display control unit 33 displays the regenerated synthetic image 103 on the display unit 13.

When the imaging process starts, imaging and camera processing are performed in the imaging device 2, and the captured images 101 are sequentially sent to the computer 3. When the CG image generation unit 53 of the computer 3 receives the captured images 101, it generates CG images 102 to be composited with each captured image 101 based on the CG file and meta information. The communication unit 44 transmits the generated CG images 102 to the imaging device 2a.

When the imaging device 2a receives the CG image 102, the CG synthesis unit 54A synthesizes the captured image 101 with the CG image 102 to generate a synthetic image 103. The display control unit 33 displays the generated synthetic image 103 on the display unit 13.

Then, the recording control unit 34 records the captured image 101 in the recording unit 24, and also associates the CG file ID, CG parameters, imaging setting information of the imaging device 2, etc. as meta information with the image data and records them in the recording unit 24 (indicated as "record" in the figure). Note that the captured image 101 recorded in the recording unit 24 may be RAW image data, image data after image processing by the camera processing unit 23, or both.

[5.2. Modification 2]
Fig. 23 is a diagram for explaining the functional configuration of a camera control unit 26B and a computer control unit 41B in the modified example 2. Fig. 24 is a block diagram illustrating an image pickup process in the modified example 2.

The camera control unit 26B in the second modification includes functional units such as a spatial information acquisition unit 31, a meta information acquisition unit 32, a display control unit 33, a recording control unit 34, and an adjustment unit 35, as well as a CG image generation unit 53B and a CG synthesis unit 54B. The computer control unit 41B in the second modification includes functional units such as a CG correction unit 51, an image acquisition unit 52, and an editing unit 55.

In Modification 2, as shown in FIG. 24, the CG image 102 and composite image 103 are not generated in the computer 3, and the CG file is recorded in the recording unit 24 of the imaging device 2. Then, the imaging device 2 generates the CG image 102 and composite image 103 based on the CG file, CG parameters, and spatial information.

Specifically, rehearsal shooting is started and shooting is started by the imaging element unit 22 (indicated as "shooting" in the figure), and various signal processing is performed by the camera process unit 23 on the image signal from the imaging element unit 22 (indicated as "camera process" in the figure). After that, the CG image generation unit 53B uses the CG file to generate a CG image 102 based on the CG parameters and spatial information (indicated as "CG generation" in the figure). In addition, the CG synthesis unit 54B synthesizes the CG image 102 with the captured image 101 to generate a synthesized image 103 (indicated as "CG synthesis" in the figure). The display control unit 33 displays the generated synthesized image 103 on the display unit 13 (indicated as "display" in the figure).

After that, when the CG parameters are adjusted (indicated as "CG parameter adjustment" in the figure), the CG image generation unit 53B regenerates the CG image 102 based on the adjusted CG parameters. Furthermore, the CG synthesis unit 54B synthesizes the CG image 102 with the captured image 101 to regenerate the synthetic image 103. The display control unit 33 displays the regenerated synthetic image 103 on the display unit 13.

When the imaging process starts, imaging and camera processing are performed in the imaging device 2, and an imaged image 101 is obtained. The CG image generating unit 53B uses the CG file to generate a CG image 102 based on CG parameters and spatial information. The CG synthesis unit 54B synthesizes the CG image 102 with the captured image 101 to generate a synthesized image 103. The display control unit 33 displays the generated synthesized image 103 on the display unit 13.

Then, the recording control unit 34 records the captured image 101 in the recording unit 24, and also records the CG file ID, CG parameters, imaging setting information of the imaging device 2, etc. as meta information in association with the image data in the recording unit 24 (indicated as "record" in the drawing). Note that the captured image 101 recorded in the recording unit 24 may be RAW image data, image data after image processing by the camera processing unit 23, or both.

<6. Summary>
According to the above embodiment, the following effects can be obtained.
The imaging device 2 of the embodiment includes an imaging unit 11 that outputs an imaged image 101, a display control unit 33 that causes a display unit 13 to display a composite image 103 generated by combining the imaged image 101 with a CG image 102 based on spatial information of the imaging space and CG parameters related to CG synthesis, and an association unit (recording control unit 34) that associates the imaged image 101 with the CG parameters.
This makes it possible for the imaging device 2 to display a composite image 103, in which the captured image 101 is composited with the CG image 102, on the display unit 13 in almost real time while the imaging unit 11 is capturing an image.
Furthermore, since the captured image 101 and the CG parameters are associated with each other without recording the composite image 103, the computer 3 can generate and edit the CG image 102 using the CG parameters in the post-processing step ST3. Therefore, the imaging device 2 allows the CG image 102 to be easily adjusted with respect to the captured image. Thus, the imaging device 2 can improve the work efficiency in video production including CG compositing.

The spatial information is associated with the CG parameters.
This allows the image capture device 2 to later correct the CG parameters based on the spatial information, or allows the correction to be performed by another device.

The associating section is the recording control section 34, which associates the captured image 101 with the CG parameters and records them.
This enables the image capturing device 2 to easily manage the captured image 101 in association with the CG parameters.

The associating section records the captured image 101 and the CG parameters in association with each other without recording the composite image 103 .
As a result, the imaging device 2 can reduce the amount of data stored in the recording unit 24 by the amount of the composite image 103 not recorded, and can also reduce the processing load.

In another device (computer 3), a composite image 103 is generated by combining a captured image 101 with a CG image 102.
This allows the imaging device 2 to reduce the processing load by having the computer 3 generate the CG image 102 and the composite image 103. Here, the processing capacity of the camera control unit 26 of the imaging device 2 is generally lower than the processing capacity of the computer control unit 41 of the computer 3.
If an image capturing device 2 with low processing power generates the CG image 102 and the composite image 103, there is a risk that the image generation will not be able to keep up with the demands of real time.
Therefore, by using a computer 3 with high processing power to generate the CG image 102 and the composite image 103, it becomes easier to ensure that the composite image 103 is displayed in real time.

The apparatus includes a CG composition unit 54A that generates a composite image 103 by combining a CG image 102 generated in another apparatus (computer 3) with a captured image 101.
This allows the imaging device 2 to have the computer 3 generate the CG image 102, thereby reducing the processing load.

The image processing apparatus includes a CG image generating section 53B that generates a CG image 102 to be composited with a captured image 101, and a CG composition section 54B that generates a composite image 103 by compositing the captured image 101 with the CG image 102.
As a result, the imaging device 2 can generate a composite image 103 by itself and display it on the display unit 13, even in an environment where the imaging device 2 and the computer 3 cannot be connected either wired or wirelessly.

The display control unit 33 causes the display unit 13 to display a composite image 103 obtained by combining a CG image 102 in real time processing in accordance with a captured image 101 being captured.
This enables the imaging device 2 to allow the cameraman 8 or the like to check the composite image 103 as a moving image in almost real time.

The image forming apparatus includes an adjustment unit 35 that adjusts CG parameters in response to user operations.
This allows the imaging device 2 to adjust CG parameters while checking a composite image 103 obtained by combining a captured image 101 captured by the imaging device 2 in a rehearsal imaging process with a CG image 102, for example.
Therefore, it is no longer necessary to strictly determine CG parameters when generating a CG file, and the effort and time required for determining CG parameters can be reduced.
Furthermore, the effort required for adjusting CG parameters in the post-processing step ST3 can be reduced.
Thus, in the imaging device 2, the working efficiency of the imaging system 1 as a whole can be improved.

The adjustment unit 35 associates the CG parameters adjusted in response to the user operation with the captured image 101 .
As a result, the CG image 102 is generated based on the adjusted CG parameters, making it possible to reduce the need for CG parameter adjustment in the post-processing step ST3.

A plurality of composite images 103, each of which is obtained by combining a CG image 102 with a captured image 101 obtained by a plurality of image capturing devices 2, are displayed on the display unit 13 in a switched manner.
This allows the cameraman 8 or the like who operates the imaging device 2a to check the CG image 102 relative to the captured image 101 obtained by another imaging device 2, such as the imaging device 2b.

A plurality of composite images 103, each of which is obtained by combining a CG image 102 with a captured image 101 obtained by a plurality of image capture devices 2, are simultaneously displayed on a display unit 13.
This allows the composite image 103 of the captured images 101 obtained by the multiple image capture devices 2 to be viewed at once using a single image capture device 2a.
Since multiple imaging devices 2 have different imaging environments such as the imaging position and the distance to the subject, it is possible that while a CG image 102 for an image 101 obtained by one imaging device 2 is preferable, a CG image 102 for an image 101 obtained by another imaging device 2 is not preferable.
Therefore, it is useful to allow the user to check the composite image 103 of the captured images 101 obtained by the multiple image capture devices 2 at once.

The imaging settings determined in any one of the plurality of imaging devices 2 are used in common by the plurality of imaging devices 2 .
This makes it easy to set up imaging for a plurality of imaging devices 2 .

The CG parameters include information regarding the start timing for starting rendering of the CG image 102, and the start timing is set based on the distance to a specific subject.
This makes it possible to start combining the CG image 102 with the captured image 101, for example, based on the position of a particular person 6 (the distance from the image capture device 2).

The CG parameters include start timing information (start timing) for starting synthesis of the CG image 102 with the captured image 101, and the start timing information is the position of a specific subject in the three-dimensional space indicated by the spatial information.
This makes it possible to start combining the CG image 102 with the captured image 101, for example, based on the position of a particular person 6 (the distance from the image capture device 2).

A CG image 102 is synthesized with each of the captured images 101 obtained by the multiple image capture devices 2 based on start timing information.
This makes it possible to synthesize a CG image 102 synchronized with the captured images 101 obtained by the multiple image capture devices 2 .

The CG parameters include information related to the rendering speed of the CG image 102, and the rendering speed is set according to the distance to a specific subject.
This makes it possible to generate a CG image 102 in which, for example, plants and the like appear in accordance with the movement of a particular person 6 and to synthesize the CG image 102 with the captured image 101 .

The camera includes a distance measuring unit (ToF sensor 12) that measures the distance to the subject, and the spatial information acquiring unit 31 acquires spatial information based on the distance to the subject measured by the distance measuring unit.
This makes it possible to easily obtain spatial information about the imaging space.

The information processing method of the embodiment outputs a captured image, and displays a composite image 103 on the display unit 13, which is generated by combining the captured image 101 with a CG image 102 based on spatial information of the captured space and CG parameters related to CG synthesis, and associates the captured image 101 with the CG parameters.

These programs can be pre-recorded on a HDD as a recording medium built into a device such as a computer device, or on a ROM in a microcomputer having a CPU. Alternatively, they can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called package software.
Such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

Furthermore, such an imaging system 1 is an imaging system 1 that includes an imaging device 2 and an information processing device (computer 3), in which the imaging device 2 includes an imaging unit 11 that outputs an imaged image 101, a display control unit 33 that causes the display unit 13 to display a composite image 103 generated by combining the imaged image 101 with a CG image 102 based on spatial information of the imaging space and CG parameters related to CG synthesis, and an association unit (recording control unit 34) that associates the imaged image 101 with the CG parameters, and the information processing device includes an editing unit 55 that adjusts the CG parameters in response to a specified user operation.
As a result, even after capturing an image, if the user wishes to modify the CG image 102, the CG image 102 can be easily edited by adjusting the CG parameters.

The editing unit 55 can align and display the timeline of the captured image 101 and the timeline of the CG image 102 based on the CG parameters.
This makes it possible to check the timelines of the captured image 101 and the CG image 102 separately.
It is also possible to separately move the timelines of the captured image 101 and the CG image 102. In other words, it is also possible to shift the synthesis timing of the CG image 102 relative to the captured image 101.

The editing unit 55 causes the display unit to display a plurality of composite images 103 in which the CG images 102 are respectively composited with the captured images 101 obtained by the plurality of image capturing devices 2 .
This allows the user to check CG images 102 corresponding to the images 101 obtained by a plurality of image capture devices 2 at once.

Note that the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

The present technology can also be configured as follows.
(1)
an imaging unit that outputs a captured image;
a display control unit that displays a composite image generated by combining the captured image with a CG image based on spatial information of the captured space and CG parameters related to CG composition on a display unit;
an association unit that associates the captured image with the CG parameters;
An imaging device comprising:
(2)
The imaging device according to any one of the preceding claims, wherein the spatial information is associated with the CG parameters.
(3)
The imaging device according to (1) or (2), wherein the associating unit is a recording control unit and associates the captured image with the CG parameters and records them.
(4)
The imaging device according to (3), wherein the associating section associates the captured image with the CG parameters and records them without recording the composite image.
(5)
The imaging device according to any one of (1) to (4), wherein the composite image is generated by combining the captured image with the CG image in another device.
(6)
The imaging device according to any one of (1) to (4), further comprising a CG synthesis unit that generates the synthesized image by synthesizing the CG image generated in another device with the captured image.
(7)
a CG image generating unit that generates the CG image to be combined with the captured image;
a CG synthesis unit that generates the synthesized image by synthesizing the CG image with the captured image;
The imaging device according to any one of (1) to (4),
(8)
The imaging device according to any one of (1) to (7), wherein the display control unit causes the display unit to display the composite image in which the CG image is composited by real-time processing corresponding to the captured image being captured.
(9)
The imaging device according to any one of (1) to (8), further comprising an adjustment unit that adjusts the CG parameters in response to a user operation.
(10)
The imaging device according to (9), wherein the adjustment unit associates the CG parameters adjusted in response to a predetermined user operation with the captured image.
(11)
The imaging device according to any one of (1) to (10), wherein the display unit switches between and displays a plurality of composite images in which the CG image is composited with the captured images obtained by the plurality of imaging devices.
(12)
The imaging device according to (11), further comprising: a display unit that displays a plurality of composite images, each of which is obtained by combining the CG image with the captured images obtained by the plurality of imaging devices, at the same time.
(13)
The imaging device according to any one of (1) to (12), wherein imaging settings determined in any one of the plurality of imaging devices are used in common in the plurality of imaging devices.
(14)
The CG parameters include information regarding start timing information for starting synthesis of the CG image with the captured image,
The imaging device according to any one of (1) to (13), wherein the start timing information is set based on a distance to a specific subject.
(15)
the CG parameters include start timing information for starting synthesis of the CG image with the captured image,
The imaging device according to any one of (1) to (14), wherein the start timing information is a position of a specific subject in a three-dimensional space indicated by the spatial information.
(16)
The imaging device according to (14) or (15), wherein the CG image is synthesized with each of the captured images obtained by the plurality of imaging devices based on the start timing information.
(17)
The CG parameters include information regarding a rendering speed of the CG image,
The imaging device according to any one of (14) and (15), wherein the drawing speed is set based on a distance to a specific subject.
(18)
Equipped with a distance measuring unit that measures the distance to the subject,
The imaging device according to any one of (1) to (16), wherein the spatial information is acquired based on a distance to the subject measured by the distance measuring unit.
(19)
Output the captured image.
displaying a composite image on a display unit, the composite image being generated by combining the captured image with a CG image based on spatial information of the captured space and CG parameters related to CG combination;
A program that causes an image capturing apparatus to execute a process of associating the captured image with the CG parameters.
(20)
An imaging system including an imaging device and an information processing device,
The imaging device includes:
an imaging unit that outputs a captured image;
a display control unit that displays a composite image generated by combining the captured image with a CG image based on spatial information of the captured space and CG parameters related to CG composition on a display unit;
an association unit that associates the captured image with the CG parameters;
Equipped with
The information processing device includes:
An imaging system comprising an editing unit that adjusts the CG parameters in response to a predetermined user operation.

Reference Signs List 1 Imaging system 2 Imaging device 3 Computer 31 Information acquisition section 32 Meta information generation section 33 Display control section 34 Recording control section 35 Adjustment section 51 CG generation section 52 Image acquisition section 53 CG synthesis section 54 Editing section

Claims

an imaging unit that outputs a captured image;
a display control unit that displays a composite image generated by combining the captured image with a CG image based on spatial information of the captured space and CG parameters related to CG composition on a display unit;
an association unit that associates the captured image with the CG parameters;
An imaging device comprising:
The imaging device according to claim 1 , wherein the spatial information and the CG parameters are associated with each other.
The imaging device according to claim 1 , wherein the associating section is a recording control section, and records the captured image and the CG parameters in association with each other.
The imaging device according to claim 3 , wherein the associating section associates the captured image with the CG parameters and records them without recording the composite image.
The imaging device according to claim 1 , wherein the composite image is generated by combining the captured image with the CG image in another device.
The imaging device according to claim 1 , further comprising a CG composition unit that generates the composite image by combining the CG image generated in another device with the captured image.
a CG image generating unit that generates the CG image to be combined with the captured image;
a CG synthesis unit that generates the synthesized image by synthesizing the CG image with the captured image;
The imaging device according to claim 1 .
The imaging apparatus according to claim 1 , wherein the display control unit causes the display unit to display the composite image obtained by combining the CG image with the captured image through real-time processing in accordance with the captured image being captured.
The imaging device according to claim 1 , further comprising an adjustment unit that adjusts the CG parameters in response to a user operation.
The imaging device according to claim 9 , wherein the adjustment section associates the CG parameters adjusted in response to a user operation with the captured image.
The imaging device according to claim 1 , wherein a plurality of composite images, each of which is obtained by combining the CG image with the captured images obtained by the plurality of imaging devices, are displayed on the display unit in a switched manner.
The imaging device according to claim 11 , wherein a plurality of composite images, each of which is obtained by combining the CG image with the images obtained by the plurality of imaging devices, are simultaneously displayed on the display unit.
The imaging device according to claim 1 , wherein imaging settings determined in any one of the plurality of imaging devices are used in common in the plurality of imaging devices.
the CG parameters include start timing information for starting synthesis of the CG image with the captured image,
The imaging device according to claim 1 , wherein the start timing information is a distance to a specific subject.
the CG parameters include start timing information for starting synthesis of the CG image with the captured image,
The imaging device according to claim 1 , wherein the start timing information is a position of a specific subject in a three-dimensional space indicated by the spatial information.
The imaging device according to claim 14 , wherein the CG image is synthesized with each of the captured images obtained by the plurality of imaging devices based on the start timing information.
The CG parameters include information regarding a rendering speed of the CG image,
The imaging device according to claim 14 , wherein the drawing speed is set based on a distance to a specific subject.
Equipped with a distance measuring unit that measures the distance to the subject,
The imaging device according to claim 1 , wherein the spatial information is acquired based on a distance to the subject measured by the distance measuring unit.
Output the captured image.
displaying a composite image on a display unit, the composite image being generated by combining the captured image with a CG image based on spatial information of the captured space and CG parameters related to CG combination;
A program that causes an image capturing apparatus to execute a process of associating the captured image with the CG parameters.
An imaging system including an imaging device and an information processing device,
The imaging device includes:
an imaging unit that outputs a captured image;
a display control unit that displays a composite image generated by combining the captured image with a CG image based on spatial information of the captured space and CG parameters related to CG composition on a display unit;
an association unit that associates the captured image with the CG parameters;
Equipped with
The information processing device includes:
An imaging system comprising an editing unit that adjusts the CG parameters in response to a predetermined user operation.