JP2023136026A - Information processing device and virtual illumination system - Google Patents

Abstract

To provide an information processing device and a virtual illumination system which can facilitate attainment of a natural illumination effect in an image of a subject.SOLUTION: An information processing device (300) comprises: a first acquisition unit which acquires image data indicating images of subjects (1, 2); a second acquisition unit which acquires illuminance distribution information showing distribution due to illuminating light illuminating the subjects from a plurality of light source positions; and a control unit which generates model information including the images of the subjects, and a characteristic in which the subjects reflect light for each position in the images, on the basis of the acquired image data and the acquired illuminance distribution information.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to an information processing device and a virtual lighting system that perform information processing regarding lighting effects of captured images.

Patent Document 1 discloses an image processing device that aims to generate a natural image with a three-dimensional effect after correcting a high brightness region of the image. This image processing device acquires normal information corresponding to the image to be corrected based on distance image data, and estimates actual illumination parameters based on the high-brightness area of the subject included in the image. ing. Furthermore, this image processing device sets the virtual illumination parameters based on the estimated actual illumination parameters so that the angle formed by the direction of the virtual illumination and the direction of the actual illumination does not become larger than a predetermined angle. Lighting processing is performed on the image based on such normal information and virtual lighting parameters.

JP2018-163648A

The present disclosure provides an information processing device and a virtual lighting system that can easily achieve natural lighting effects in an image of a subject.

An information processing device according to an aspect of the present disclosure includes a first acquisition unit that acquires image data representing an image of a subject, and a second acquisition unit that acquires illuminance distribution information that represents the distribution of illumination light irradiated onto the subject from a plurality of light source positions. The apparatus includes an acquisition unit and a control unit that generates model information including an image of a subject and characteristics of light reflection of the subject for each position in the image, based on the acquired image data and illuminance distribution information.

An information processing device according to another aspect of the present disclosure includes a storage unit that stores model information including an image of a subject, an operation unit that receives a user operation regarding virtual illumination for the subject, and an information processing device configured according to the user operation. The present invention includes a control unit that controls a simulation process that calculates a lighting effect for model information according to virtual illumination, and a display unit that displays an image to which the lighting effect has been applied by the simulation process. The model information indicates the characteristic of light reflection by the subject for each position in the image of the subject, based on the distribution of illumination light irradiated onto the subject from a plurality of light source positions. The control unit performs simulation processing to apply a lighting effect to the image of the subject based on the characteristics of the subject indicated by the model information.

An information processing device according to still another aspect of the present disclosure acquires image data indicating an image of a subject captured by an imaging device and/or illuminance distribution information indicating a distribution of illumination light irradiated onto a subject from a lighting device. and a control unit that controls the imaging device and the lighting device. The imaging device sequentially images a subject at a predetermined period to generate image data. The control unit causes the illumination device to irradiate the subject with illumination light when the imaging device is not imaging the subject in a predetermined period, and causes the imaging device to irradiate the subject with illumination light when the illumination device is not irradiating the subject with illumination light. to be imaged.

A virtual illumination system in the present disclosure includes an imaging device that images a subject and generates image data, an illumination device that irradiates the subject with illumination light and obtains information indicating distribution due to the illumination light, and any of the above information processing. and a device.

According to the information processing device and virtual lighting system according to the present disclosure, it is possible to easily realize a natural lighting effect in an image of a subject.

Diagram for explaining a virtual lighting system according to Embodiment 1 of the present disclosure Diagram showing the configuration of a digital camera in a virtual lighting system A diagram illustrating the configuration of a lighting device in a virtual lighting system. Front view illustrating the structure of the lighting device A diagram illustrating the configuration of an information terminal in a virtual lighting system Flowchart illustrating the object photographing operation in the virtual lighting system A diagram illustrating an environment for object photographing operation in a virtual lighting system Diagram illustrating the data structure of model information in a virtual lighting system Flowchart for explaining reflection characteristic estimation processing in a virtual lighting system Diagram illustrating illuminance distribution data by a lighting device of a virtual lighting system Diagram to explain the operation of the lighting device of the virtual lighting system Diagram illustrating a reflectance map in virtual illumination image processing Diagram for explaining coordinate transformation in virtual lighting image processing Flowchart illustrating background photography operations in a virtual lighting system A diagram illustrating the environment of background shooting operation in the virtual lighting system Flowchart illustrating placement simulation processing in a virtual lighting system Diagram showing a display example of placement simulation processing in a virtual lighting system Diagram for explaining the calculation of lighting effects in virtual lighting image processing Flowchart illustrating a video shooting operation in the virtual lighting system according to Embodiment 2 Timing chart for explaining an example of exclusive control operation in a virtual lighting system A diagram illustrating modification example 1 of the article photographing operation in the virtual illumination system. A diagram illustrating a second modification of the article photographing operation in the virtual illumination system.

Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, detailed explanations of well-known matters and redundant explanations of substantially the same configurations may be omitted. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the claimed subject matter.

(Embodiment 1)
1. Configuration A virtual lighting system according to Embodiment 1 of the present disclosure will be described using FIG. 1.

The virtual lighting system 10 according to this embodiment includes, for example, as shown in FIG. 1, a digital camera 100, a lighting device 200, and an information terminal 300. This system 10 is an image synthesis simulation that visualizes the virtual placement of an item 1 such as furniture desired by a customer in a background 2 of a separately desired room, etc., in an interior shop or a real estate company, for example. Applicable to simulation. The article 1 and the background 2 are examples of subjects that are each photographed, for example, at separate photographing sites 15.

In such a placement simulation, even if the captured image of the article 1 and the captured image of the background 2 are combined, there may be a problem that the combined image becomes unnatural due to inconsistencies in illuminance or the like of the photographing site 15, which is different between the two. Therefore, in this system 10, when photographing the article 1 etc. with the digital camera 100, the illumination device 200 is used to perform image processing of the illumination effect after the fact, and the above-mentioned mismatch is corrected in the composite image during the placement simulation. Achieve a natural lighting effect that eliminates the problem.

The configuration of each device 100, 200, 300 in this system 10 will be described below.

1-1. Configuration of Digital Camera The configuration of the digital camera 100 in this embodiment will be described using FIG. 2.

FIG. 2 is a diagram illustrating the configuration of the digital camera 100 in the present system 10. The digital camera 100 is an example of an imaging device in this embodiment. The digital camera 100 of this embodiment includes an image sensor 115, an image processing engine 120, a display monitor 130, and a control section 135. Further, the digital camera 100 includes a buffer memory 125, a card slot 140, a flash memory 145, an operation section 150, and a communication module 155. Further, the digital camera 100 includes, for example, an optical system 110 and a lens driving section 112.

The optical system 110 includes a focus lens, a zoom lens, an optical image stabilization lens (OIS), an aperture, a shutter, and the like. The focus lens is a lens for changing the focus state of the subject image formed on the image sensor 115. A zoom lens is a lens for changing the magnification of a subject image formed by an optical system. Each of the focus lenses and the like is composed of one or more lenses.

The lens driving section 112 drives the focus lens and the like in the optical system 110. The lens driving section 112 includes a motor, and moves the focus lens along the optical axis of the optical system 110 based on the control of the control section 135. The configuration for driving the focus lens in the lens drive unit 112 can be realized using a DC motor, a stepping motor, a servo motor, an ultrasonic motor, or the like.

The image sensor 115 captures a subject image formed through the optical system 110 and generates image data. The image data constitutes image data representing an image captured by the image sensor 115. Image sensor 115 generates a new frame of image data at a predetermined frame rate (eg, 30 frames/second). The generation timing of image data and the electronic shutter operation in the image sensor 115 are controlled by the control unit 135. As the image sensor 115, various image sensors such as a CMOS image sensor, a CCD image sensor, or an NMOS image sensor can be used.

The image sensor 115 performs a still image imaging operation, a through image imaging operation, and the like. The through image is mainly a moving image, and is displayed on the display monitor 130 for the user to decide on a composition for capturing a still image. A through image and a still image are each an example of a captured image in this embodiment. The image sensor 115 is an example of an imaging unit in this embodiment.

The image processing engine 120 performs various processes on the imaged data output from the image sensor 115 to generate image data, performs various processes on the image data, and generates an image to be displayed on the display monitor 130. or generate. Various types of processing include, but are not limited to, white balance correction, gamma correction, YC conversion processing, electronic zoom processing, compression processing, and expansion processing. The image processing engine 120 may be configured with a hard-wired electronic circuit, or may be configured with a microcomputer, processor, etc. using a program.

The display monitor 130 is an example of a display unit that displays various information. For example, the display monitor 130 displays an image (through image) represented by image data captured by the image sensor 115 and subjected to image processing by the image processing engine 120. The display monitor 130 also displays a menu screen and the like for the user to make various settings for the digital camera 100. Display monitor 130 can be configured with, for example, a liquid crystal display device or an organic EL device.

The operation unit 150 is a general term for hard keys such as operation buttons and operation levers provided on the exterior of the digital camera 100, and accepts operations by the user. The operation unit 150 includes, for example, a release button, a mode dial, and a touch panel. When the operation unit 150 receives an operation by the user, it transmits an operation signal corresponding to the user operation to the control unit 135.

The control unit 135 centrally controls the operation of the digital camera 100 as a whole. The control unit 135 includes a CPU and the like, and the CPU executes a program (software) to realize a predetermined function. The control unit 135 may include a processor configured with a dedicated electronic circuit designed to implement a predetermined function, instead of the CPU. That is, the control unit 135 can be realized by various processors such as a CPU, MPU, GPU, DSP, FPGA, and ASIC. The control unit 135 may be composed of one or more processors. Further, the control unit 135 may be configured with one semiconductor chip together with the image processing engine 120 and the like.

The buffer memory 125 is a recording medium that functions as a work memory for the image processing engine 120 and the control unit 135. The buffer memory 125 is realized by DRAM (Dynamic Random Access Memory) or the like. Flash memory 145 is a nonvolatile recording medium. Further, although not shown, the control unit 135 may have various types of internal memories, such as a built-in ROM. Various programs executed by the control unit 135 are stored in the ROM. Further, the control unit 135 may include a built-in RAM that functions as a work area for the CPU.

The card slot 140 is a means into which a removable memory card 142 is inserted. A memory card 142 can be electrically and mechanically connected to the card slot 140. The memory card 142 is an external memory that includes a recording element such as a flash memory inside. Memory card 142 can store data such as image data generated by image processing engine 120.

The communication module 155 is a module (circuit) that connects to external equipment according to a predetermined communication standard for wired or wireless communication. The predetermined communication standards include, for example, USB, HDMI (registered trademark), IEEE802.11, Wi-Fi, Bluetooth, and the like. Digital camera 100 can communicate with other devices via communication module 155.

1-2. Configuration of Lighting Device The configuration of the lighting device 200 in this embodiment will be explained using FIGS. 3 and 4.

FIG. 3 is a diagram illustrating the configuration of the lighting device 200 in the present system 10. The lighting device 200 includes, for example, as shown in FIG. 3, a light emitting section 210, a light receiving section 220, a control section 230, a storage section 240, and a communication section 250. The structure of the lighting device 200 is illustrated in FIG.

FIG. 4 illustrates a front view of the illumination device 200 viewed from the direction in which illumination light is emitted. The lighting device 200 of the present system 10 is configured in various types, such as a ceiling-mounted type or a stand type. The lighting device 200 may be a clip-on type that is attached to the digital camera 100, or may be configured integrally with a photographing box that accommodates a subject.

In the lighting device 200, the light emitting unit 210 is configured with various light source devices so as to emit illumination light having various wavelengths in the visible region, for example. The light emitting unit 210 includes a plurality of illumination light sources 211 to 213, as shown in FIG. 4, for example. FIG. 4 shows an example in which the light emitting section 210 is composed of three illumination light sources 211 to 213. The light emitting unit 210 of the lighting device 200 is not limited to this, and may include, for example, four or more illumination light sources 211 to 213. Hereinafter, the illumination light sources 211 to 213 will be collectively referred to as the illumination light source 21.

The illumination light source 21 is, for example, a variable wavelength LED, and emits illumination light of multiple colors such as RGB at a predetermined light distribution angle. The illumination light sources 21 may each include a plurality of monochrome LEDs or the like. The light distribution angle of the illumination light source 21 is set such that illumination light is irradiated onto a spatial range including the subject 11 at a photographing site 15 for photographing the subject 11 such as the article 1 or the background 2, for example. They are configured to be able to emit illumination light having the same wavelength. The light emitting unit 210 may emit white light from the illumination light source 21 as illumination light.

For example, the first to third illumination light sources 211 to 213 in the light emitting section 210 are arranged so as to surround the light receiving section 220 in the lighting device 200. Due to this arrangement, the light emitting unit 210 has different light source positions in the first to third illumination light sources 211 to 213. The first to third illumination light sources 211 to 213 are arranged in a direction to irradiate illumination light onto a common spatial range, respectively.

The light receiving section 220 has a light receiving surface configured to be able to receive light in a visible region corresponding to the illumination light from the light emitting section 210. The light receiving unit 220 is composed of various image sensors such as a CCD image sensor, a CMOS image sensor, or an NMOS image sensor. The light receiving unit 220 captures a monochrome image, for example, and generates data indicating a spatial distribution of illuminance as a result of light reception. The light receiving unit 220 is not limited to monochrome, and may include color filters such as RGB. The light receiving unit 220 may output data of images of each color as light reception results for each color light.

The control unit 230 controls the overall operation of the lighting device 200, for example. Control unit 230 includes, for example, a CPU or MPU that cooperates with software to implement predetermined functions. The control unit 230 may be a hardware circuit such as a dedicated electronic circuit or a reconfigurable electronic circuit designed to implement a predetermined function. The control unit 230 may be configured with various semiconductor integrated circuits such as a CPU, MPU, microcomputer, DSP, FPGA, and ASIC.

The storage unit 240 includes a ROM and a RAM that store programs and data necessary to realize the functions of the lighting device 200. The storage unit 240 stores, for example, information indicating the positional relationship between each of the illumination light sources 211 to 213 and the light receiving unit 220.

The communication unit 250 is a module (circuit) that connects to external equipment according to a predetermined communication standard for wired or wireless communication. The predetermined communication standards include, for example, USB, HDMI, IEEE802.11, Wi-Fi, Bluetooth, and the like.

1-3. Configuration of Information Terminal The configuration of the information terminal 300 in this embodiment will be described using FIG. 5.

FIG. 5 is a diagram illustrating the configuration of the information terminal 300. The information terminal 300 is an example of an information processing device configured with, for example, a personal computer, a smartphone, or a tablet terminal. The information terminal 300 illustrated in FIG. 5 includes a control section 310, a storage section 320, an operation section 330, a display section 340, and a communication section 350.

Control unit 310 includes, for example, a CPU or MPU that implements predetermined functions in cooperation with software. The control unit 310 controls the overall operation of the information terminal 300, for example. The control unit 310 reads data and programs stored in the storage unit 320 and performs various arithmetic operations to realize various functions.

The control unit 310 executes, for example, a program including a group of instructions for realizing each of the above functions. The above program may be provided from a communication network such as the Internet, or may be stored in a portable recording medium. Furthermore, the control unit 310 may be a hardware circuit such as a dedicated electronic circuit or a reconfigurable electronic circuit designed to implement each of the above functions. The control unit 310 may be configured with various semiconductor integrated circuits such as a CPU, MPU, GPU, GPGPU, TPU, microcomputer, DSP, FPGA, and ASIC.

The storage unit 320 is a storage medium that stores programs and data necessary to implement the functions of the information terminal 300. The storage unit 320 includes a storage unit 321 and a temporary storage unit 322, as shown in FIG.

The storage unit 321 stores parameters, data, control programs, etc. for realizing predetermined functions. The storage unit 321 is composed of, for example, an HDD or an SSD. For example, the storage unit 321 stores the above programs, various image data, and the like.

The temporary storage unit 322 is configured with a RAM such as DRAM or SRAM, and temporarily stores (that is, holds) data. For example, the temporary storage unit 322 holds image data that is being edited. Further, the temporary storage section 322 may function as a work area of the control section 310, or may be configured as a storage area in the internal memory of the control section 310.

The operation unit 330 is a general term for operation members operated by the user. The operation unit 330 may be, for example, a keyboard, a mouse, a touch pad, a touch panel, a button, a switch, or the like. The operation unit 330 also includes various GUIs such as virtual buttons, icons, cursors, and objects displayed on the display unit 340.

The display unit 340 is an example of an output unit configured with, for example, a liquid crystal display or an organic EL display. The display unit 340 may display various types of information such as various GUIs for operating the operation unit 330 and information input from the operation unit 330.

The communication unit 350 is a module (circuit) that connects to external equipment according to a predetermined communication standard for wired or wireless communication. The predetermined communication standards include, for example, USB, HDMI, IEEE802.11, Wi-Fi, Bluetooth, and the like. The communication unit 350 may connect the information terminal 300 to a communication network such as the Internet. The communication unit 350 is an example of an acquisition unit that receives various information from an external device or a communication network.

The configuration of the information terminal 300 as described above is an example, and the configuration of the information terminal 300 is not limited to this. For example, the acquisition unit in the information terminal 300 may be realized by cooperation with various software in the control unit 310 and the like. The input unit in the information terminal 300 acquires various information by reading out various information stored in various storage media (for example, the storage unit 321) into the work area (for example, the temporary storage unit 322) of the control unit 310. There may be.

Moreover, various display devices such as a projector and a head-mounted display may be used for the display unit 340 of the information terminal 300. Further, for example, when an external display device is used, the display section 340 of the information terminal 300 may be an output interface circuit for outputting video signals or the like based on, for example, the HDMI standard.

2. Operation The operation of the virtual lighting system 10 configured as described above will be described below.

In this system 10, the digital camera 100 photographs the article 1 to be subjected to placement simulation in advance, for example. In such an article photographing operation, the present system 10 acquires information using the illumination device 200. Furthermore, in this embodiment, a photographing operation using the digital camera 100 and the illumination device 200 is also performed for the background 2 of the placement simulation. Based on the above photographic results, the present system 10 reproduces the state in which the article 1 is placed on the background 2 according to the user's operation in a placement simulation, and provides a lighting effect of virtual illumination.

The details of the operation of this system 10 will be described below. Although an example of the operation of the system 10 using the information terminal 300 will be described below as an example of the operation of the present system 10, various photographing operations and placement simulation processing may be performed without using the same terminal.

2-1. Article Photographing Operation An example of article photographing operation in the present system 10 will be explained using FIGS. 6 to 8. In this operation example, the digital camera 100 photographs a moving image or the like over the entire circumference of the article 1, which is an example of a subject, and the illumination device 200 acquires information. Thereby, in the subsequent placement simulation, the orientation of the article 1 can be changed over its entire circumference.

FIG. 6 is a flowchart illustrating the article photographing operation in the present system 10. FIG. 7 illustrates an environment for the article photographing operation of FIG. 6. Each process in the flowchart illustrated in FIG. 6 is executed by, for example, the control unit 310 of the information terminal 300.

In this operation example, as shown in FIG. 7, for example, a rotating table 400 is used that rotates the object 1 to be photographed. The rotary table 400 includes, for example, a stage on which the article 1 is placed, a motor, etc., and is configured to rotate to various angular positions in one rotation from 0 degrees to 360 degrees corresponding to the entire circumference of the article 1. . Further, in this operation example, the information terminal 300 is connected to each device 100, 200, 400 for data communication, and controls the operation of each device 100, 200, 400. Further, although FIG. 7 illustrates a stand-type lighting device 200, the present invention is not particularly limited thereto.

For example, as shown in FIG. 6, the control unit 310 of the information terminal 300 transmits a control signal to the lighting device 200 to cause the lighting device 200 to perform operations of emitting and receiving illumination light (S1). The control unit 310 acquires, for example, via the communication unit 350, illuminance distribution data (see FIG. 10) regarding the illumination light irradiated onto the article 1 from the lighting device 200.

For example, in the lighting device 200, the control unit 230 sequentially causes the first to third illumination light sources 211 to 213 to emit light in the light emitting unit 210 in response to a control signal from the information terminal 300, and in the light receiving unit 220 in a time-sharing manner. The reflected light from each of the illumination light sources 211 to 213 is received. The control unit 230 of the lighting device 200 generates illuminance distribution data indicating the light reception result for each illumination light source 21, and transmits the illuminance distribution data to the information terminal 300 via the communication unit 250. The generation of illuminance distribution data is performed for each color, such as RGB. For example, the lighting device 200 sequentially emits one type of illumination light among the colored illumination lights from each illumination light source 21 in each step S1.

Further, the control unit 310 transmits a control signal to the digital camera 100 to cause the digital camera 100 to perform a shooting operation for each frame of a moving image, for example (S2). The information terminal 300 acquires video data output from the digital camera 100, for example, via the communication unit 350. In this embodiment, the photographing operation of the digital camera 100 may be performed while the illumination device 200 is emitting light, or may be performed while the light emission is stopped. Model information of the article 1 may be generated using captured images of the article 1 illuminated with various types of illumination light.

For example, in the digital camera 100, the control unit 135 controls the exposure timing of the image sensor 115 according to a control signal from the information terminal 300, and sequentially captures frame images at each step S2. For example, a frame image captured in one step S2 shows the appearance of the article 1 as seen from the digital camera 100 in an orientation corresponding to the angular position of the rotary table 400. For example, the digital camera 100 outputs moving image data from the communication module 155 as an example of image data as a result of such imaging. Note that the digital camera 100 may perform continuous still image shooting instead of moving image shooting, or may repeatedly take single shots.

For example, the control unit 310 determines whether the illumination device 200 has emitted and received light for all types of illumination light while the rotating table 400 is at a specific angular position (S3). All types of illumination light include illumination light of each color from each of the first to third illumination light sources 211 to 213. When the illumination device 200 is not performing all types of light emission/reception operations (NO in S3), the control unit 310 changes the type of illumination light and repeats the processing of steps S1 to S3.

When the lighting device 200 performs all types of light emitting and receiving operations (YES in S3), for example, the control unit 310 causes the article 1 to reflect various types of illumination light based on the illuminance distribution data acquired from the lighting device 200. Arithmetic processing for estimating characteristics is performed (S4). In the reflection characteristic estimation process (S4), for example, estimation information indicating the estimation results of the reflection characteristics, such as the normal direction of various positions on the surface of the article 1 and the reflectance of each color light, is obtained from various angles at which the article 1 is imaged. Generated for each location. Details of the process in step S4 will be described later.

Further, the control unit 310 controls the rotary table 400 to change the angular position by a predetermined rotation angle, for example (S5). The rotation angle in step S5 is appropriately set (for example, 15 degrees) from the viewpoint of updating the portion imaged by the digital camera 100 of the external appearance of the article 1 on the rotating table 400 for each frame. Rotation of the turntable 400 is performed intermittently, for example, every step S3.

For example, the control unit 310 determines whether or not the rotation table 400 has completed the entire circumference of the article 1 (eg, 360 degree rotation) based on the changed angular position (S6). If the rotation by the rotary table 400 is not completed (NO in S6), the control unit 310 performs the processing from step S2 onwards again for the new angular position. By repeating steps S1 to S6, video data and illuminance distribution data of the photographed results over the entire circumference of the article 1 are acquired.

For example, when the rotation by the turntable 400 is completed (YES in S6), the control unit 310 uses the video data acquired from the digital camera 100 and the estimated information on the reflection characteristics generated at each step S4. Model information of article 1 is created based on (S7). The model information of the article 1 is an example of first model information indicating a model of the article 1 (object) used in the placement simulation, for example, in a three-dimensional shape. The data structure of model information in this system 10 is illustrated in FIG.

For example, as shown in FIG. 8, the model information of the article 1 in this example includes a captured image showing the appearance of the article 1 over the entire circumference of the article 1, and the reflection characteristics of the article 1 on the captured image. The reflection characteristics include characteristics in which light is reflected at different positions on the captured image. For example, the control unit 310 associates normal directions and reflectances estimated at various positions in each frame image corresponding to different angular positions in the video data. The control unit 310 may cut out and process a region in which the article 1 is shown from each frame image. Such model information may be managed in various data structures, for example, as video data and meta information for each frame thereof. Furthermore, a three-dimensional image obtained by combining a plurality of captured images and a reflectance model in which reflection characteristics are associated with each position in the three-dimensional shape of the article 1 may be managed.

The control unit 310 stores, for example, the generated model information (S7) of the article 1 in the storage unit 320, and ends the processing of this flow.

According to the above article photographing operation, the digital camera 100 photographs the entire circumference of the article 1, and at the same time, illuminance distribution data is acquired by the illumination device 200 (S1 to S6), and virtual illumination around the entire circumference of the article 1 is obtained. Model information for realizing this can be obtained (S7).

In steps S1 and S2 above, an example has been described in which the timing of operations such as video shooting is controlled by the information terminal 300, but the timing does not need to be particularly controlled by the information terminal 300. For example, one of the digital camera 100 and the lighting device 200 may control the operation timing of the other, and the devices 100 and 200 may be communicatively connected to each other to transmit and receive control signals.

Further, in step S5 above, an example has been described in which the information terminal 300 controls the rotation of the turntable 400, but the information terminal 300 does not need to control the rotation. For example, the rotating table 400 may rotate in response to an instruction from the digital camera 100 or the lighting device 200, a user operation, or the like. Alternatively, a rotating table 400 that is manually rotated by a user may be used. The information terminal 300 may acquire information indicating the angular position from the rotary table 400 or may estimate the angular position from an image captured by the digital camera 100. The rotation of the turntable 400 is not limited to intermittent rotation, but may be continuous.

A plurality of frame images captured while repeating steps S1 to S3 may be used to generate a composite image as a result of photography by the digital camera 100. For example, the digital camera 100 performs various types of image synthesis such as depth synthesis, HDR (high dynamic range) synthesis, or super-resolution synthesis based on a plurality of frame images from such specific angular positions to create a composite image. May be generated. Such a composite image may be used as the model information in step S7.

Further, in the above explanation, an example was explained in which the digital camera 100 and the lighting device 200 are operated simultaneously in the article photographing operation (FIG. 6), but the article photographing operation does not have to be performed simultaneously. . For example, the entire circumference of the article 1 may be photographed using the digital camera 100, and the light emitting/receiving operation may be separately performed by the illumination device 200 for the entire circumference of the article 1. The information terminal 300 can appropriately input the information acquired by each device 100, 200 in this way and perform the processing from step S4 onward in FIG.

2-1-1. Reflection Characteristic Estimation Process The reflection characteristic estimation process in step S4 of FIG. 6 will be explained using FIGS. 9 to 13. FIG. 9 is a flowchart for explaining the reflection characteristic estimation process (S4) in the present system 10.

First, the control unit 310 of the present system 10 refers to the illuminance distribution data acquired in step S1, which is repeated in steps S1 to S3 in FIG. 6, for example (S11). Such illuminance distribution data is, for example, illuminance distribution data of each light reception result corresponding to the emission of illumination light of each color from each of the first to third illumination light sources 211 to 213 when the position of the object such as the article 1 is the same. including.

FIG. 10 illustrates illuminance distribution data D1 to D3 for one color by the illumination device 200 of the present system 10. FIG. 11 is a diagram for explaining the operation of the lighting device 200 when generating the illuminance distribution data D1 of FIG. 10.

FIG. 11 shows an example of the optical path of illumination light emitted by the illumination device 200 at the shooting site 15. In the example of FIG. 11, the illumination light emitted from the first illumination light source 211 in the light emitting unit 210 at the emission intensity L1 is emitted in the light direction _S1 , and is reflected at the surface position Pa of the subject 11. The reflected light of such illumination light is observed at a light receiving position pa in the light receiving unit 220 at an illuminance I that depends on the normal direction N to the surface position Pa of the subject 11 and the reflection coefficient ρ.

The light ray direction _S1 and the normal direction N are each represented by a unit vector in a three-dimensional space, and arrows indicating vectors are attached in the figure. The light beam direction _S1 is set in advance for each illumination light source 21, for example, and is a known setting value. The normal direction N and the reflection coefficient ρ for each surface position Pa of the subject 11 are unknown targets for estimation. The illumination device 200 of this embodiment obtains the illuminance I1 of the reflected light of the illumination light from the first illumination light source 211 for each light receiving position pa, and shows the distribution of the illuminance I1 over the entire light receiving surface of the light receiving unit 220. Illuminance distribution data D1 is generated.

FIG. 10A illustrates illuminance distribution data D1 as a result of light reception according to light emission from the first illumination light source 211. FIG. 10(B) illustrates illuminance distribution data D2 according to light emission from the second illumination light source 212, and FIG. 10(C) illustrates illuminance distribution data D3 according to light emission from the third illumination light source 213. In FIGS. 10(A) to 10(C), the magnitude of the illuminance is shown by shading so that the gray becomes lighter as the illuminance increases.

The illuminance distribution data D1 illustrated in FIG. The illuminance distribution of reflected light reflected by Pa is shown. In such an illuminance distribution, the illuminance at each surface position Pa of the subject 11 is distributed to the corresponding light receiving position pa in the coordinate system on the light receiving surface of the light receiving unit 220 of the lighting device 200, that is, the light receiving coordinate (x, y).

In the illuminance distribution data D2 of FIG. 10(B), based on the illumination light irradiated from the second illumination light source 212 instead of the first illumination light source 211 to the spatial range of the same environment as FIG. 10(A), An illuminance distribution different from that in FIG. 10(A) is obtained. Similarly, the illuminance distribution data D3 in FIG. 10(C) has an illuminance distribution different from that in FIGS. 10(A) and (B) based on the illumination light from the third illumination light source 213. For example, the illuminance observed at the same light receiving position pa between FIGS. 10(A) to 10(C) differs depending on the normal direction N of the surface position Pa of the corresponding object 11 and the reflection coefficient ρ. .

The present system 10 analyzes the differences between the illuminance distribution data D1 to D3 caused by the plurality of illumination light sources 211 to 213 using, for example, the photometric stereo method, and thereby calculates the method for each surface position Pa of the subject 11 at the shooting site 15. Estimate the line direction N and reflection coefficient ρ. Note that the normal direction N has two variables of freedom as an estimation target due to the unit vector normalization condition for three-dimensional vector components.

For example, the control unit 310 in the present system 10 refers to the illuminance distribution data D1 to D3 of each illumination light source 211 to 213 (S11) and creates a reflectance map using two variables p and q in the normal direction N. (S12). FIG. 12 illustrates reflectance maps M1 to M3 corresponding to each of the illumination light sources 211 to 213.

FIG. 12(A) illustrates a reflectance map M1 corresponding to the illuminance distribution data D1 of FIG. 10(A). FIG. 12(B) illustrates a reflectance map M2 corresponding to FIG. 10(B), and FIG. 12(C) illustrates a reflectance map M3 corresponding to FIG. 10(C). The reflectance maps M1 to M3 indicate the distribution of the proportion of reflected light of illumination light from a specific light source position being received, for example in the pq plane, and include contour lines regarding the brightness of the reflected light. The pq plane is defined by a coordinate system of two variables p and q corresponding to various normal directions N, for example, by Gaussian mapping.

The control unit 310 determines the normal direction N (i.e., Arithmetic processing is performed to estimate the surface inclination) and the reflection coefficient ρ (S13). The control unit 310 calculates, for example, the following equations (1) to (3) assuming a Lambertian surface.

In the above equations (1) to (3), Ln represents the emission intensity of the illumination light from the n-th illumination light sources 211 to 213 (n=1, 2, 3). Sn indicates the direction of the light beam from the n-th illumination light sources 211 to 213 as a unit vector. In indicates the illuminance received as the illuminance distribution data D1 to D3 by the n-th illumination light sources 211 to 213. Further, in the above equations (1) to (3), "x" indicates multiplication, and "." indicates an inner product between vectors. The above equations (1), (2), and (3) correspond to reflectance maps M1, M2, and M3 in FIGS. 10(A) to (C), respectively.

In the above equations (1) to (3), for example, the three variables p and q in the normal direction N and the reflection coefficient ρ are to be estimated, so the first to third illumination light sources 211 to Based on the information on the light reception result obtained by 213, estimated values of each variable are determined. For example, the control unit 310 executes the calculations of equations (1) to (3) above for each light receiving position pa at the light receiving coordinates (x, y) (S13).

Next, the control unit 310 converts the estimation information indicating the estimation results of the normal direction N of the subject 11 and the reflection coefficient ρ from the light receiving coordinates (x, y) to the imaging coordinates (X, Y). Then, estimated information on reflection characteristics is generated (S14). The imaging coordinates (X, Y) are a coordinate system indicating the position on the captured image indicated by the image data to be corrected, and indicate the position on the imaging surface of the image sensor 115 of the digital camera 100. 13A and 13B show the light receiving coordinates (x, y) and the imaging coordinates (X, Y) before and after the conversion in step S14, respectively.

The coordinate transformation in step S14 is calculated by, for example, triangulation according to the positional relationship between the digital camera 100 and the lighting device 200 at the shooting site 15 in the present system 10. At this time, the focal length of the image taken by the digital camera 100 or the focal length of the light received by the illumination device 200 may be used as the subject distance. For example, the control unit 310 acquires various types of information for coordinate transformation in this system 10 in advance in steps S1, S2, etc. in FIG. 6, and executes the process in step S14.

For example, when the control unit 310 generates reflection characteristic estimation information for the corresponding frame image in the video data (S14), it ends the reflection characteristic estimation process (S4 in FIG. 6) and proceeds to step S7.

According to the above reflection characteristic estimation process (S4), the illuminance distribution data D1 to D3 obtained using the plurality of illumination light sources 211 to 213 in the illumination device 200 in the photographing environment of the object such as the article 1, The normal direction N and reflection coefficient ρ can be estimated (S13).

For example, in step S4, which is repeated in steps S1 to S6 in FIG. 6, the control unit 310 generates estimated information on the reflection characteristics for the corresponding angular positions. For example, the control unit 310 generates the illuminance distribution data for the angular position by performing the same processing as steps S11 to S14 based on all types of illuminance distribution data for each angular position. Alternatively, the control unit 310 uses the initially generated reflection characteristic estimation information and the illuminance distribution data of each step S1 after the initial stage to correct the initial reflection characteristic estimation information. Estimated information regarding frame images may be generated sequentially.

In step S13 above, calculation equations (1) to (3) for estimating the photometric stereo method are illustrated. In this system 10, the estimation calculation in step S13 is not limited to the above equations (1) to (3), and various changes are possible. For example, the number of illumination light sources 21 may be increased to four or more, and the number of simultaneous arithmetic expressions may be increased.

In addition, although the above equations (1) to (3) use the assumption of a Lambertian surface, various calculation formulas may be applied that are not limited to this assumption, and AI techniques such as machine learning models may be applied. Good too. The control unit 310 can apply a calculation similar to the estimation calculation in step S13 to calculate the lighting effect of the virtual illumination according to various estimation methods.

In step S14 above, an example in which triangulation is used for coordinate conversion between the light reception coordinates (x, y) of the illumination device 200 and the imaging coordinates (X, Y) of the digital camera 100 has been described. The processing in step S14 is not particularly limited to the above, and distance measurement may be performed in the digital camera 100 using a DFD (Depth From Defocus) or the like, for example. Additionally, a distance measuring sensor such as a TOF (Time Of Flight) sensor may be used in the present system 10. Further, various image recognition techniques may be used to estimate the mutual positional relationship from, for example, images corresponding to the illuminance distribution data D1 to D3 of the lighting device 200 and images captured by the digital camera 100. Step S14 may be realized using various AI techniques such as such a machine learning model for image recognition.

Furthermore, in the above description, an example has been described in which the reflection characteristic estimation process (S4 in FIG. 6) is performed by the information terminal 300 while repeating steps S1 to S6. The process of step S4 is not particularly limited to this, and may be performed, for example, after repeating steps S1 to S6.

Further, the reflection characteristic estimation process (S4) may be performed in the lighting device 200. The lighting device 200 may transmit reflection characteristic estimation information to the information terminal 300 as an example of the illuminance distribution information instead of the illuminance distribution data.

2-2. Background Photographing Operation An example of the operation of acquiring information about the background 2 by the illumination device 200 and photographing an image by the digital camera 100 in the present system 10 will be described with reference to FIGS. 14 and 15. An example of the operation of the system 10 using still image shooting will be described below.

FIG. 14 is a flowchart illustrating a background photographing operation in the present system 10. FIG. 15 illustrates an environment for the background photographing operation of FIG. 14. Each process in the flowchart illustrated in FIG. 14 is executed by, for example, the control unit 310 of the information terminal 300.

In this operation example, as shown in FIG. 15, for example, the digital camera 100 is fixedly placed on a tripod or the like. Furthermore, the lighting device 200 is of a ceiling-mounted type as an example. Each device 100, 200 does not have to be the same device used specifically for the article photographing operation (FIG. 7). In this operation example as well, it is assumed that the information terminal 300 is connected to each device 100, 200 so as to be capable of data communication.

For example, the control unit 310 of the information terminal 300 causes the lighting device 200 to perform a light emitting and receiving operation (S21), similar to step S1 in FIG. In step S21, the control unit 310 acquires all types of illuminance distribution data for each color of illumination light from the first to third illumination light sources 211 to 213, for example, from the illumination device 200.

Further, for example, the control unit 310 causes the digital camera 100 to perform a still image shooting operation (S22). The control unit 310 acquires image data of a photographed result from the digital camera 100. The processing order of steps S21 and S22 is not particularly limited. Further, in steps S21 and S22, the control unit 310 does not need to particularly control the operation timing of each device 100, 200, and may acquire each data of the operation result of each device 100, 200 as appropriate.

Further, for example, the control unit 310 performs a process of estimating the reflection characteristics of the background 2 based on the illuminance distribution data from the illumination device 200 (S23). In step S23, the control unit 310 performs processing similar to steps S11 to S14 in FIG. 9, for example, to generate estimated information on the reflection characteristics for the background 2.

Next, the control unit 310 creates model information for the background 2 based on the image data obtained by the digital camera 100 and the estimated information on the reflection characteristics for the background 2, for example, similar to step S7 in FIG. 6 (S24). ). The model information for the background 2 is an example of second model information indicating the model for the background 2 in the placement simulation, similar to the model information for the article 1. The model information of the background 2 indicates, for example, the normal direction and reflectance estimated for each position in the captured image of the background 2 in association with each other in two dimensions based on the imaging coordinates of the digital camera 100. For example, model information for background 2 may be managed as still image data and its meta information.

The control unit 310 stores, for example, the generated model information of the background 2 (S24) in the storage unit 320, and ends the processing of this flow.

According to the above-described background photographing operation, it is possible to obtain model information that associates a captured image with estimated information on reflection characteristics using the digital camera 100 and the lighting device 200 for the room, etc. used as the background 2 of the placement simulation. (S21 to S24).

In the above description, an example of the operation in which still image shooting (S22) is performed once in the background shooting operation has been described, but still image shooting may be performed multiple times, or video shooting may be performed. . For example, in the example of FIG. 15, a plurality of captured images may be acquired according to different orientations while rotating the digital camera 100 at a fixed position. In this case, regarding the model information of the background 2 as well, a plurality of captured images and their respective reflection characteristics may be managed in a three-dimensional shape.

Furthermore, in the above description, the background photographing operation using the illumination device 200 has been described. In this system 10, the lighting device 200 may not be particularly used for the background 2, and only the digital camera 100 may take an image. For example, the control unit 310 may estimate the reflection characteristics of the background 2 regarding the ambient light based on the captured image of the background 2 using an image estimation method.

Further, in the above description, an example has been described in which the photographing result by the digital camera 100 is used as the model information of the background 2, but the present invention is not particularly limited to this. For example, the model information of the background 2 may include a two-dimensional or three-dimensional image generated by CG (computer graphics) or the like and information on reflection characteristics. The following placement simulation process can also be performed using the background 2 model information. Also, regarding the model information of the article 1, a CG image may be used instead of the photographed result of the article 1. On the other hand, by using the photographed results of the article 1 or the background 2, it is possible to improve the reality of the texture of each subject in the simulated image.

2-3. Placement Simulation Process The operation of performing a placement simulation in the virtual lighting system 10 of this embodiment using the model information of the article 1 and the background 2 obtained as described above will be described with reference to FIGS. 16 to 18.

FIG. 16 is a flowchart illustrating the placement simulation process in the present system 10. FIG. 17 shows a display example of the placement simulation process of FIG. 16.

Each process shown in the flowchart of FIG. 16 is started, for example, with various model information stored in the storage unit 320 of the information terminal 300, and is executed by the control unit 310.

First, the control unit 310 causes the display unit 340 to display a screen for performing a placement simulation for the article 1 selected in advance (S31). A display example in step S31 is shown in FIG.

FIG. 17 shows a display example of a simulation screen in the present system 10. For example, the simulation screen displayed on the display unit 340 of the information terminal 300 includes a simulation image field 41, a background selection field 42, and a lighting setting field 43. The simulation image column 41 is a column for visualizing the placement simulation of the article 1. For example, the article image G1 generated by the model information of the article 1 and the background image G2 generated by the model information of the background 2 are displayed. Display various images included.

The control unit 310 receives various user operations via the operation unit 330 while displaying a simulation screen as illustrated in FIG. 17 on the display unit 340, for example (S32 to S34). At the time of step S31, the simulation image column 41 may display the article image G1 without particularly displaying the background image G2.

The control unit 310 sets the background in the layout simulation, for example, in response to a user operation in the background selection field 42 (S32). The background selection field 42 includes, for example, a thumbnail image for accepting a user's operation to select a desired background from among a plurality of preset background candidates. For example, the control unit 310 reads out the model information of the background 2 selected by the user's operation from the storage unit 320, and uses the captured image in the read model information of the background 2 as the background image G2 in the simulation image field 41 as the article image. Display it superimposed on G1.

Further, the control unit 310 sets the arrangement of the article 1, for example, according to the user's operation in the simulation image column 41 (S33). In step S33, a user operation can be input to, for example, shift the position of the article image G1 with respect to the background image G2 in the simulation image column 41 or rotate the orientation of the article image G1. For example, in response to the user's rotation operation, the control unit 310 extracts a frame image corresponding to the user's desired orientation from the frame images for the entire circumference in the model information of the article 1, and selects the frame image in the simulation image column 41 as the article image G1. to be displayed.

Further, the control unit 310 sets virtual illumination indicating the environmental light in the layout simulation, for example, in response to a user operation in the illumination setting field 43 (S34). For example, the lighting setting field 43 accepts user operations for setting lighting parameters such as the intensity of virtual environmental light, color temperature, and light distribution angle. The control unit 310 calculates, for example, based on the model information of the background 2, lighting parameters and estimated values of the light source position regarding the environmental light when the background image G2 is captured. The estimated value of the lighting parameter is used as an initial value in the lighting setting field 43, for example.

Next, based on the background 2, the arrangement of the article 1, and the virtual illumination settings (S32 to S34) as described above, the control unit 310 displays the natural image of the article 1 placed on the background 2 in the simulation image field 41. Image processing of the virtual illumination is performed to provide a lighting effect (S35). The process of step S35 will be explained using FIG. 18.

In FIG. 18, for example, virtual illumination Ea indicating environmental light is illustrated in a three-dimensional coordinate system (X, Y, Z) corresponding to the imaging coordinates (X, Y) of each captured image in each model information. The control unit 310 estimates the light source position of the virtual illumination Ea from the background 2 according to the settings in steps S32 and S33 in FIG. Identify.

The control unit 310 calculates the illumination effect of the virtual illumination, for example, as in the following equation (10), based on the various parameters La, Sa of the virtual illumination Ea, the normal direction N of the estimation result of the subject 11, and the reflection coefficient ρ. (S35).

In the above equation (10), Io indicates the brightness of the original image before correction in step S35, and Ie indicates the brightness of the corrected image, and each is a (monochromatic) pixel value, for example. The control unit 310 performs the calculation of the above equation (10) for each pixel position in the imaging coordinates (X, Y), for example. In this way, the control unit 310 generates a corrected image indicating the correction result of adding the lighting effect of the virtual illumination to the original image (S35).

For example, in step S35, the control unit 310 uses the article image G1 of the rotated frame of the user operation in the model information of the article 1 as the original image, and applies the lighting effect according to the above equation (10). Further, for example, when the lighting parameters of the virtual lighting Ea are changed from the estimated value (initial value) of the environment light in the model information of the background 2 by a user operation, the control unit 310 changes the lighting parameters of the background image G2 based on the changes caused by the user operation. A lighting effect is also applied to the area in the same way as above. The control unit 310 generates a corrected image by superimposing the article image G1 with the lighting effect and the background image G2 in accordance with the set arrangement of the article 1, and causes the corrected image to be displayed in the simulation image field 41.

Returning to FIG. 16, the control unit 310 receives various user operations while displaying a corrected image as a result of virtual illumination image processing (S35) on the simulation screen, and inputs, for example, an operation to end the placement simulation. It is determined whether or not it has been done (S36).
For example, when the user performs an operation to change various settings in each column 41 to 43 of the simulation screen, the control unit 310 proceeds to NO in step S36, and executes the processing from step S32 onwards to reflect the changed settings. Do it again. Thereby, each time the user changes the settings, the control unit 310 calculates a natural lighting effect under the changed settings and visualizes it to the user as a corrected image in the simulation image field 41, for example (S35).

When an operation to end the placement simulation is input (YES in S36), the control unit 310 stores, for example, image data representing a corrected image finally obtained in virtual illumination image processing (S35) in the storage unit as a simulation result. 320 (S37).

The control unit 310 ends the placement simulation process of FIG. 16, for example, by recording the image data of the simulation result (S37).

According to the above arrangement simulation processing, the arrangement of the article image G1 and the background image G2 desired by the user is determined based on information obtained using the illumination device 200 when photographing the article 1 and the background 2 in the present system 10. A natural lighting effect can be achieved (S35).

In step S32 above, an example has been described in which the user is asked to select the background 2 with the article 1 set in advance in the placement simulation process. The placement simulation process of this embodiment is not particularly limited to this, and for example, the information terminal 300 may accept a user operation to select the article 1 to be placed on the background 2 with the background 2 set in advance. Further, a plurality of articles 1 placed on the background 2 may be selectable. For example, the information terminal 300 can perform such a placement simulation process in the same manner as described above by storing the model information of each article 1 in the storage unit 320 in advance.

In step S<b>34 above, an example has been described in which a user operation for adjusting the illumination parameters of the ambient light is accepted in the illumination setting field 43 . The information terminal 300 of the present system 10 may further receive a user operation to adjust the light source position of the ambient light. For example, the user may input the lighting equipment to be placed in the background 2 room, or the position and direction of sunlight entering from the window of the room, depending on the desired time period. The information terminal 300 can set virtual lighting according to such user input (S34) and calculate a natural lighting effect for the virtual lighting desired by the user (S35).

Further, the above arrangement simulation process may be performed using VR (virtual reality) or AR (augmented reality) technology. For example, instead of or in addition to FIG. 17, the article 1 and the background 2 may be displayed three-dimensionally. Even in this case, lighting effects such as ambient light are calculated in advance using three-dimensional model information based on information obtained using the illumination device 200 when photographing with the digital camera 100 (S35). Natural lighting effects can be given to dimensional composite images. Further, for example, when applying AR technology, a real background may be used as the background 2.

Furthermore, in the above arrangement simulation process, the lighting device 200 does not have to be used for both the model information of the article 1 and the model information of the background 2, and the lighting device 200 can be used only for one of the model information. good. Even in this case, the reflection characteristics of the subject can be reproduced with high precision with respect to model information using the illumination device 200, making it easier to obtain natural lighting effects.

3. Summary As described above, in this embodiment, the information terminal 300, which is an example of an information processing device, includes the communication unit 350, which is an example of first and second acquisition units, and the control unit 310. The first acquisition unit acquires image data representing an image of a subject, such as an image of the subject such as article 1, captured by the digital camera 100, which is an example of an imaging device (S1). The second acquisition unit acquires illuminance distribution data as an example of illuminance distribution information indicating the distribution of illumination light irradiated onto the subject from a plurality of light source positions of the illumination device 200 (S2). Based on the acquired image data and illuminance distribution information, the control unit 310 generates model information including a captured image of the subject and a characteristic of the subject reflecting light such as visible light for each position in the captured image ( S7).

According to the information terminal 300 described above, the virtual lighting system 10 realizes a natural lighting effect in the image of the object using the model information generated based on the image data of the object obtained by the digital camera 100 and the illuminance distribution information of the object obtained by the illumination device 200. It can be made easier. The image of the subject is not limited to the image captured by the digital camera 100, but may be an image generated based on the captured image, or may be a CG image.

In the information terminal 300 of this embodiment, the first acquisition unit acquires image data indicating a plurality of captured images in which the subject is captured from different directions, or one captured image (S2, S22). The model information indicates an image of the subject based on such a captured image. That is, the image of the subject in the model information is composed of the above-mentioned captured image. For example, the image of the subject may be the captured image itself, an image obtained by performing various image processing on the captured image, or a composite image of multiple captured images. By using such actual captured images, the reality of the subject can be improved in the image synthesis simulation. The image of the subject does not necessarily have to be based on a captured image, and may be a CG image.

In this embodiment, the model information includes a plurality of captured images and the characteristics of the subject for each position in each captured image (S7, see FIG. 7). As a result, natural lighting effects can be applied to images of objects in a plurality of orientations, making it easier to perform placement simulations and the like in the virtual lighting system 10.

In the information terminal 300 of this embodiment, the model information includes an image of the subject over at least a portion of the entire circumference of the subject (S6, S7, see FIG. 7). That is, the image of the subject in the model information is an image covering at least part of the entire circumference of the subject. Thereby, in the placement simulation of the virtual lighting system 10, a natural lighting effect can be obtained over the entire circumference of the subject or a portion thereof.

In the information terminal 300 of the present embodiment, the characteristic that the object reflects light in the model information includes at least one of the normal direction N and the reflectance ρ of the object for each position in the captured image (S4, FIG. reference). The lighting effect can be calculated by reflecting the reflection characteristics of the object, making it easier to realize natural lighting effects in the image of the object.

In the present embodiment, the second acquisition unit acquires illuminance distribution information obtained by the illumination device 200 that individually emits illumination light from a plurality of mutually different light source positions, for example, by the first to third illumination light sources 211 to 213. . The illuminance distribution information includes a plurality of illuminance distributions in which reflected light of each illumination light from a plurality of light source positions is received, respectively (see FIG. 10). Using such a lighting device 200, it is possible to easily achieve a natural lighting effect in an image of a subject.

The information terminal 300 in this embodiment includes a storage section 320, an operation section 330, a control section 310, and a display section 340. The storage unit 320 stores model information including an image of a subject. The operation unit 330 receives user operations regarding virtual illumination of the subject (S32 to S34). The control unit 310 controls a placement simulation process as an example of a simulation process that calculates a lighting effect for model information according to virtual lighting that is set in accordance with a user operation (S31 to S37). The display unit 340 displays, for example, in a simulation image column 41 (FIG. 17), an image to which a lighting effect has been added by the simulation process of the control unit 310. For example, the model information of the article 1 indicates the reflection characteristics for each position in the image of the article 1 based on illuminance distribution data when the illumination light is irradiated onto the article 1 from the illumination device 200. The control unit 310 performs simulation processing to apply a lighting effect to the image of the subject based on the reflection characteristics of the subject indicated by the model information (S35). According to this information terminal 300, using the model information, it is possible to easily realize a natural lighting effect in an image of a subject in a layout simulation of the present system 10, etc.

In the information terminal 300 of this embodiment, the model information stored in the storage unit 320 includes model information of the article 1 (first model information) and model information of the background 2 (second model information). At least one of the first and second model information indicates the reflection characteristics of the corresponding subject of the article 1 (object) and the background 2 based on the distribution of illumination light from the illumination device 200. The operation unit 330 receives a user operation (S33) that specifies the arrangement of the article 1 in the background 2. The control unit 310 sets virtual illumination to indicate the environmental light in the background 2, and performs image processing for the placement simulation (S35). Thereby, a natural lighting effect can be given to the article image G1 as the background 2 environmental light.

In the information terminal 300 of this embodiment, the operation unit 330 accepts a user operation to adjust a lighting parameter, which is an example of a parameter related to virtual lighting (S34). The control unit 310 performs image processing so as to apply a lighting effect to the image of the subject according to the adjusted lighting parameters (S35). Thereby, it is possible to easily realize a natural lighting effect in the image of the subject regarding the virtual lighting desired by the user.

In the information terminal 300 of the present embodiment, the model information of the article 1 generated by the article photographing operation shown in FIG. 1 shows an image of article 1 over at least a portion thereof; In response to a user operation to change the orientation of the subject on the operation unit 330 (S33), the control unit 310 applies a lighting effect to the image of the article 1 in the changed orientation and displays it on the display unit 340 (S35). In this way, the model information can be used to achieve natural lighting effects for images of objects in multiple orientations.

In the information terminal 300 of this embodiment, the model information is generated for each position in the captured image based on the distribution of illumination light irradiated from the illumination device 200 to the subject in the shooting environment of the captured image, for example, through reflection characteristic estimation processing (S4). shows the characteristic of the subject reflecting light. Such model information makes it easier to achieve natural lighting effects in the image of the subject.

In this embodiment, the virtual illumination system 10 includes a digital camera 100 that captures an image of a subject and generates image data, a lighting device 200 that irradiates the subject with illumination light and obtains information indicating the distribution of the illumination light, and information and a terminal 300. According to the present system 10, the information acquired by the digital camera 100 and the lighting device 200 can be used in the information terminal 300 to facilitate the realization of natural lighting effects in the image of the subject.

(Embodiment 2)
Embodiment 2 of the present disclosure will be described below with reference to FIGS. 19 and 20. In the first embodiment, an example in which the virtual lighting system 10 is applied to a placement simulation has been described. In the second embodiment, an example in which the virtual lighting system 10 is applied to video production will be described.

Hereinafter, the virtual illumination system 10 according to the present embodiment will be described, omitting the description of the same configuration and operation as the virtual illumination system 10 according to the first embodiment as appropriate.

FIG. 19 is a flowchart illustrating the operation of the virtual lighting system 10 according to the second embodiment. The process shown in the flowchart of FIG. 19 starts, for example, when a user operation instructs to start shooting a moving image.

In this embodiment, the virtual lighting system 10 is applied to a use in which a user such as a photographer produces a video work such as a moving image in a desired direction. This system 10 uses the lighting device 200 when shooting a video with the digital camera 100, thereby making it possible to add lighting effects after the fact using the information terminal 300 or the like. According to the present system 10, it is possible to realize video production with a high degree of freedom in which desired lighting effects can be edited in post-processing without the need to prepare particularly difficult lighting equipment.

In the video shooting operation of the present system 10 (FIG. 19), for example, the control unit 310 transmits a control signal to cause the illumination device 200 to perform a light emission/reception operation for each type of illumination light (S1A). Further, for example, the control unit 310 transmits a control signal to the digital camera 100 to cause the digital camera 100 to perform a shooting operation for each frame of the moving image (S2A).

In steps S1A and S2A of this embodiment, for example, the control unit 310 causes the lighting device 200 and the digital camera 100 to operate exclusively in the same process as steps S1 and S2 of the first embodiment (FIG. 6). Generate each control signal. Such exclusive control will be described later (see FIG. 20).

Further, for example, the control unit 310 performs reflection characteristic estimation processing for each frame photographed by the digital camera 100 (S4A). The process in step S4A is performed in the same way as the reflection characteristic estimation process in the first embodiment (S4 in FIG. 6) based on the illuminance distribution data acquired by the illumination device 200 in a predetermined number of frames before the current one.

For example, when the end of video shooting is not instructed (NO in S10), the control unit 310 repeats the processing from step S1A onward. For example, when an instruction to end video shooting is input by a user operation (YES in S10), the control unit 310 appropriately associates the captured video data with the estimated information of the reflectance for each frame and stores it in the storage unit 240, etc. The process shown in the flowchart of FIG. 19 ends.

According to the above process, by operating the illumination device 200 simultaneously with video shooting by the digital camera 100, illuminance distribution data that changes moment by moment according to the camera work or movement of the subject in the video being shot can be obtained. be able to. At this time, there may be a problem that the light emission for obtaining the illuminance distribution data is reflected in the video data, resulting in unnatural lighting effects and the like. Exclusive control for solving these new problems in the present system 10 will be explained using FIG. 20.

FIG. 20 is a timing chart for explaining an example of exclusive control operation in this system 10. FIG. 20(A) illustrates the light emission timing of the lighting device 200. FIG. 20(B) illustrates the exposure timing of the digital camera 100.

In the example of FIG. 20, an exposure mask period T11 and an exposure period T12 are provided in the frame period T1 of the moving image. For example, the exposure mask period T11 corresponds to step S1, and the exposure period T12 corresponds to step S2. For example, if the exposure mask period T11 is set to 3 milliseconds for a frame period T1 of 1/60 seconds, the exposure period T12 can be secured to about 14 milliseconds. Note that the periods T1, T11, and T12 are not limited to those described above, and can be set as appropriate.

In this operation example, the illumination device 200 performs an operation of emitting and receiving illumination light during an exposure mask period T11 in a frame period T1 (S1A), as shown in FIG. 20A, for example. During the subsequent exposure period T12, the illumination device 200 stops emitting illumination light, and generates illuminance distribution data based on the light reception result during the exposure mask period T11.

For example, the illumination device 200 emits one type of light of each color in the first to third illumination light sources 211 to 213 during each exposure mask period T11 (S1A). The illumination device 200 may emit and receive multiple types of light during the exposure mask period T11. For example, in the above operation example, the illumination device 200 may emit and receive all types of light in the initial stage of video shooting.

On the other hand, as shown in FIG. 20B, for example, the digital camera 100 does not expose the image sensor 115 during the exposure mask period T11 in which the illumination light is emitted in the frame period T1, and the emission of the illumination light is stopped. The image sensor 115 is exposed during the exposure period T12 (S2A).

In this system 10, as described above, by exclusively controlling the exposure during video shooting of the digital camera 100 and the light emission of the illumination device 200, it is possible to obtain illuminance distribution data during video shooting, and also to obtain the illuminance distribution data. It is possible to avoid a situation where the light emission affects the video. Therefore, it is possible to easily realize a natural lighting effect for a captured image of a moving subject or the like in a moving image.

As described above, the information terminal 300, which is an example of the information processing device in this embodiment, uses image data indicating an image of a subject captured by the digital camera 100 and the distribution of illumination light irradiated onto the subject from the lighting device 200. The communication unit 350 is an example of an acquisition unit that acquires the illuminance distribution information shown in FIG. The digital camera 100 generates image data by sequentially capturing images of a subject at a predetermined period. The control unit 310 causes the illumination device 200 to irradiate the subject with illumination light when the digital camera 100 is not capturing an image of the subject in a predetermined period, and when the illumination device 200 is not irradiating the subject with illumination light, The digital camera 100 is caused to image the subject (S1A, S2A, see FIG. 20).

Even with such an information processing device, it is possible to easily realize a natural lighting effect in the image of the subject using the virtual lighting system 10. Further, such an information processing device may acquire either image data or illuminance distribution information. For example, the digital camera 100 or the lighting device 200 may be an example of the information processing device.

(Other embodiments)
As described above, Embodiments 1 and 2 have been described as examples of the technology disclosed in this application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments in which changes, replacements, additions, omissions, etc. are made as appropriate. Furthermore, it is also possible to create a new embodiment by combining the components described in each of the above embodiments.

In the first embodiment described above, the article photographing operation using the rotary table 400 has been described, but the article photographing operation may also be performed without using the rotary table 400. Such a modification will be explained using FIG. 21.

FIG. 21 illustrates the environment of Modified Example 1 of the article photographing operation. In this modification, instead of the turntable 400, a rail 450 that surrounds the article 1 on the same plane, such as a horizontal plane, is arranged. The rail 450 is configured to be slidable on the support portion of the digital camera 100. In this modification, in the same operation as in FIG. 6, instead of rotating the article 1 using the rotary table 400, the digital camera 100 is rotated around the entire circumference of the article 1 along the rail 450 to take a video or the like. With this also, it is possible to obtain captured images of the entire circumference of the article 1 and create model information of the article 1 in the same way as in the above example.

Furthermore, instead of using the rail 450, the object photographing operation as described above may be performed by a photographer using the digital camera 100 moving around the object 1 while taking the photograph. Further, in the article photographing operation, a plurality of illumination devices 200 may be used. For example, a plurality of lighting devices 200 may be arranged around the article 1.

Further, in each of the embodiments described above, the article photographing operation is described over the entire circumference of the article 1, but the photographing operation does not necessarily have to be performed over the entire circumference of the article 1. For example, the digital camera 100 may take a video of a portion of the entire circumference of the article 1, or may take still images, etc. of the article 1 multiple times from different directions. . This also allows model information to be obtained for a plurality of images of the article 1 taken from different directions, which can be used for placement simulation.

Further, the entire circumference of the article 1 or a portion thereof in the above-described article photographing operation is not limited to being on the same plane, but may be three-dimensional. Such a modification will be explained using FIG. 22.

FIG. 22 is a diagram for explaining a second modification of the article photographing operation. In this modification, a slider 455 that allows the digital camera 100 to slide three-dimensionally around the article 1 is used instead of the rail 450 of the above modification. The slider 455 has the same configuration as the rail 450, for example, and also has a rotation axis capable of three-dimensional rotation to remove the slide movement range of the digital camera 100 from a horizontal plane or the like. In the present embodiment, in the article photographing operation (FIG. 6) of the digital camera 100, for example, the slider 455 may be used to photograph a desired range around the entire circumference of the article 1 on the entire spherical surface or hemispherical surface. As a result, the three-dimensional modeling in the model information of the article 1 (S7 in FIG. 6) can be made highly accurate, and the direction in which the orientation of the article 1 can be changed when the user views the article 1 in a placement simulation etc. ( 17)).

Further, in each of the embodiments described above, an example of model information of the article 1 and the like has been described, but the model information is not limited to this. In the virtual lighting system 10 of the present embodiment, the model information may be information in which image data obtained by capturing an image of a subject using the digital camera 100 and illuminance distribution data of the subject obtained by the illumination device 200 are associated with each other. For example, in a process similar to the flow of FIG. 6, the control unit 310 of the information terminal 300 may generate model information based on the above association instead of step S7, without particularly performing the process of step S4. Based on such model information, a process similar to step S4 may be performed, for example, when executing the placement simulation process (FIG. 16).

Further, in each of the above embodiments, an example has been described in which various information processing of the virtual lighting system 10 is performed by the information terminal 300. The present system 10 is not particularly limited to this, and may further include, for example, a server device that communicates data with the information terminal 300. For example, the virtual lighting calculation process (S35) in the placement simulation process (FIG. 16) may be performed by the server device. The information terminal 300 may transmit the information obtained in steps S32 to S34 to such a server device, and may receive the processing result in step S35 from the server device. Also, in various photographing operations, part or all of the reflection characteristic estimation process (S4, S23) and the generation of various model information (S7, S24) may be performed by the server device.

Furthermore, in each of the above embodiments, the virtual illumination system 10 that performs placement simulation processing of the article 1 on the background 2 has been described. In this embodiment, the virtual illumination system 10 may perform arithmetic processing regarding virtual illumination, which is not limited to placement simulation. For example, the virtual lighting system 10 of this embodiment may perform virtual lighting simulation processing for the user to check the appearance of the article 1. Even in such a case, it may be difficult to reproduce the appearance such as the color or texture of the actual article 1 during simulation.

On the other hand, according to the virtual lighting system 10 of this embodiment, natural lighting effects can be realized by using information obtained using the lighting device 200 when photographing the article 1 in the same manner as in the above embodiment. , the reproducibility of the appearance of the article 1 can be improved. For example, in the virtual illumination system 10 of this embodiment, the control unit 310 can omit display of the background image G2 and the like in the same process as in FIG. 16, and perform the simulation process of this embodiment.

Further, the virtual illumination system 10 of this embodiment is not limited to the simulation process for only the article 1, but may be used for the simulation process for only the background 2. For example, the real estate property may be photographed using the lighting device 200 as the background 2, and the system 10 may perform a virtual preview simulation process using the model information obtained in this manner.

Further, in each of the above embodiments, the virtual lighting system 10 including the digital camera 100, the lighting device 200, and the information terminal 300 has been described, but the virtual lighting system 10 of the present embodiment is not limited to this. For example, in the virtual lighting system 10 of this embodiment, the digital camera 100 may be omitted. For example, an image of the subject may be captured or acquired using an image sensor or the like as the light receiving section 220 of the illumination device 200. Further, in the present system 10, the illumination device 200 and the imaging device may be configured integrally. Furthermore, the image of the subject may be acquired from outside the system 10.

Furthermore, in each of the above embodiments, the information terminal 300 is illustrated as an example of an information processing apparatus. In this embodiment, the information processing device is not particularly limited to the information terminal 300, and may be, for example, a server device. Further, the digital camera 100 may have the function of an information processing device using the control unit 135 or the like. In this case, the image sensor 115 of the digital camera 100 may function as an example of an imaging device.

Furthermore, in each of the above embodiments, the digital camera 100 includes the optical system 110 and the lens driving section 112. The imaging device of this embodiment does not need to include the optical system 110 and the lens drive unit 112, and may be, for example, an interchangeable lens type camera.

Further, in each of the above embodiments, a digital camera has been described as an example of an imaging device, but the invention is not limited to this. The imaging device of the present disclosure may be any electronic device (for example, a video camera, a smartphone, a tablet terminal, etc.) that has an image capturing function.

As described above, the embodiments have been described as examples of the technology in the present disclosure. To that end, the accompanying drawings and detailed description have been provided. Therefore, among the components described in the attached drawings and detailed description, there are not only components that are essential for solving the problem, but also components that are not essential for solving the problem, in order to exemplify the above technology. may also be included.

The present disclosure is applicable to applications in which lighting effects are subsequently added to images of various subjects.

10 virtual lighting system 100 digital camera 200 lighting device 210 light emitting unit 220 light receiving unit 300 information terminal 310 control unit 320 storage unit 330 operation unit 340 display unit 350 communication unit

Claims (10)

a first acquisition unit that acquires image data representing an image of the subject;
a second acquisition unit that acquires illuminance distribution information indicating a distribution of illumination light irradiated onto the subject from a plurality of light source positions;
An information processing apparatus comprising: a control unit that generates model information including an image of the subject and characteristics of light reflection of the subject for each position in the image, based on acquired image data and illuminance distribution information. The first acquisition unit acquires image data indicating one or more captured images of the subject,
The image of the subject in the model information is composed of the captured image,
The information processing device according to claim 1. The information processing apparatus according to claim 1, wherein the image of the subject is an image covering at least part of the entire circumference of the subject. The information processing apparatus according to claim 1, wherein the characteristic that the subject reflects light includes at least one of a normal direction and reflectance of the subject for each position in the image. The second acquisition unit acquires the illuminance distribution information obtained by a lighting device that individually emits the illumination light from a plurality of mutually different light source positions,
The information processing apparatus according to claim 1, wherein the illuminance distribution information includes a plurality of illuminance distributions in which reflected light of each illumination light from the plurality of light source positions is received. a storage unit that stores model information including an image of the subject;
an operation unit that accepts a user operation regarding virtual illumination for the subject;
a control unit that controls a simulation process that calculates a lighting effect for the model information according to virtual lighting that is set according to the user operation;
a display unit that displays an image to which the lighting effect is applied by the simulation process;
The model information indicates a characteristic of the subject reflecting light for each position in the image of the subject based on the distribution of illumination light irradiated to the subject from a plurality of light source positions,
The control unit is an information processing device that performs the simulation process to apply the lighting effect to the image of the subject based on the characteristics of the subject indicated by the model information. The model information includes first model information indicating an object as the subject and second model information indicating a background,
At least one of the first and second model information indicates characteristics of a corresponding subject among the object and the background based on the distribution by the illumination light,
The operation unit accepts a user operation specifying a placement of the object in the background,
The information processing apparatus according to claim 6, wherein the control unit performs the simulation process by setting the virtual illumination to indicate environmental light in the background. The model information indicates an image of the subject over at least a portion of the entire circumference of the subject;
7. The control unit is configured to apply the lighting effect to an image of the subject in the changed orientation and display the image on the display unit in response to a user operation to change the orientation of the subject on the operation unit. information processing equipment. an acquisition unit that acquires image data indicating an image of a subject captured by an imaging device and/or illuminance distribution information indicating a distribution of illumination light irradiated onto the subject from a lighting device;
comprising a control unit that controls the imaging device and the lighting device,
The imaging device sequentially images the subject at a predetermined period to generate the image data,
In the predetermined period, the control unit:
When the imaging device is not imaging the subject, irradiating the subject with the illumination light from the illumination device,
An information processing device that causes the imaging device to image the subject when the illumination device does not irradiate the subject with the illumination light. an illumination device that irradiates a subject with illumination light and obtains information indicating a distribution due to the illumination light;
A virtual lighting system comprising the information processing device according to any one of claims 1 to 9.

Images