JP2009081822A - Data transmission device and method, and view environment control apparatus, system and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a view environment control system capable of achieving illumination control in an optimal view environment corresponding to a camera work, illumination situation and scene setting situation when photographing a display video image. <P>SOLUTION: A data transmitting device divides video data into segments each constituted of one or more frames of an arbitrary number, adds segment division information indicating the starting time and/or the section of each segment, and at least any one of camera work information in photographing the video data, photographing illumination information, and scene setting information to video data in the unit of a segment and transmits the video data. On the other hand, a data receiving device separates from the video data at least any one of the segment division information, camera work situation, the photographing illumination information and the scene setting information and produces illumination control data for controlling illumination light of an illuminator 38 using these pieces of information. Thus, illumination light in a view environment can be appropriately controlled while being adapted to contents of a display video image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a data transmission apparatus and data capable of controlling illumination light around a video display device in accordance with the atmosphere and scene setting of the video shooting scene when the video is displayed on the video display device. The present invention relates to a transmission method, a viewing environment control device, a viewing environment control system, and a viewing environment control method.

  For example, when displaying an image with an image display device such as a television receiver, or when projecting and displaying an image using a projector device, the surrounding illumination light is adjusted according to the display image. There is known a technique for providing a viewing effect such as enhancing the sense of presence.

  For example, Patent Document 1 calculates the mixed light intensity ratio of the three primary colors of the light source for each frame from the color signal (RGB) and luminance signal (Y) of the display image of a color television, and performs dimming control in conjunction with the image. A light color variable illumination device is disclosed. This variable-light-color illuminating device extracts a color signal (RGB) and a luminance signal (Y) from a display image of a color television, and uses the three color lights (red light, green) used as a light source from the color signal and the luminance signal. Light, blue light) is calculated, the illuminance of the three-color light is set according to the illuminance ratio, and the three-color light is mixed and output as illumination light.

  Further, for example, Patent Document 2 discloses a video effect lighting apparatus that performs illumination control around a division unit by dividing a television image into a plurality of portions and detecting an average hue of the corresponding division unit. Yes. This video effect lighting device includes illumination means for illuminating the surroundings of the installation location of the color television, divides the video displayed on the color television into a plurality of parts, and divides the video corresponding to the portion illuminated by the illumination means The average hue of the part is detected, and the illumination means is controlled based on the detected hue.

Furthermore, for example, Patent Document 3 does not simply calculate the average chromaticity and average luminance of the entire screen of the image display device, but instead of pixels of skin color portions such as human faces from the image displayed on the screen of the image display device. The remaining portion is considered as the background portion, and only the RGB signal and luminance signal of each pixel in the background portion are taken out to obtain the average chromaticity and average luminance, and the chromaticity and luminance of the back wall of the image display device are A method of controlling illumination so as to be the same as the average chromaticity and average luminance of the entire screen or the background portion excluding human skin color is disclosed.
Japanese Patent Laid-Open No. 2-158094 JP-A-2-253503 JP-A-3-184203

  In the conventional viewing environment control apparatus described above, the feature amount (color signal and luminance signal) for each frame (screen) in the video signal to be displayed is detected and the illumination light is controlled. It is difficult to generate illumination light that matches the field (atmosphere) of the image. For example, if you illuminate the surrounding area with improper color illumination light due to the influence of the clothes of the subject person included in the video signal or artifacts in the background of the subject, the atmosphere of each scene can be reproduced. , Can not maintain the sense of reality for each scene. In other words, viewing environment lighting that deviates significantly from the lighting situation at the time of shooting a video scene, on the contrary, impairs the sense of reality.

  The above-mentioned Patent Document 3 controls the illumination light in the viewing environment space based on the feature amount (color signal and luminance signal) of each pixel in the background portion with the remaining portion from which the skin color portion pixels are removed as the background portion. However, for example, in the case of an image where the ground and buildings occupy a lot in the background part, it will irradiate surroundings with illumination light of an inappropriate color due to this influence, so on the contrary, the presence and atmosphere Will be lost. That is, in order to estimate the lighting condition (atmosphere) at the time of shooting the display image from the video signal, it is not sufficient to remove the skin-colored pixels, and only the background pixels representing the lighting condition (atmosphere) at the time of shooting are appropriate. It is necessary to extract and use.

  In the above-described conventional technology, the illumination light state changes according to the change of the luminance and hue of the video signal for each frame, and particularly when the degree of change in luminance and hue between frames is large. The problem is that the light changes complicatedly and the viewer feels uncomfortable with the flicker. Furthermore, it is not preferable that the illumination light fluctuates in accordance with changes in luminance and hue for each frame during the display of one scene in which the illumination state at the time of shooting does not change, which adversely inhibits the atmosphere for each scene. .

  FIG. 37 is a diagram for explaining an example of the problem of the illumination control according to the conventional technique, and shows a part of continuous moving images. In the example shown in FIG. 37, a scene of a video shot with a scene setting of outdoors in a sunny day is created. This scene consists of a video obtained by a series of camera work shooting without switching the camera. In this example, an image of a skier sliding down from above the camera toward the vicinity of the camera is taken. Skiers are dressed in red and the sky is clear.

  In other words, this video scene was shot under certain camera work conditions such as camera position, angle, number of subjects, camera movement, and camera lens type: low positive, low angle, one shot (1S), fixed, standard lens. It has been done.

  The video of this scene has a large blue sky area in the initial frame, and the skier's red clothing area gradually increases as the skier slides down and approaches the camera. That is, as the video in the scene progresses, the ratio of the colors constituting each frame changes. That is, the subject size is that the frames A to D are log shots and the frame E is a full figure.

  In such a case, when the illumination light is controlled using the chromaticity and luminance for each frame, the blue light changes from strong illumination light to red illumination light. In other words, even though the video was shot under constant natural light illumination, illumination light that does not take into account the lighting conditions at the time of shooting is generated and irradiated. Will give a sense of incongruity. In addition, if the color of the illumination light changes in a single segment of scenes in which one scene setting (atmosphere) is continuous, the atmosphere of the scene is still disturbed and the viewer feels uncomfortable.

  FIG. 38 is a diagram for explaining another example of the problem of the illumination control according to the conventional technique. In the example shown in FIG. 38, a scene of an image shot with a scene setting of “moonlit night outdoors” is created. This scene is composed of three shots (1, 2, 3) with different camera work. In Shot 1, the camera takes a long shot of the target ghost. Then, when switching to shot 2, the ghost is shot with a bust shot. In shot 3, the camera returns to the camera position of shot 1 again. These shots are intended and configured as a segment of a scene in which one atmosphere continues even though the camera work is different.

  In other words, the camera work situation used for shooting this scene is that the camera position, angle, camera movement, and camera lens type are eye height, horizontal angle, fixed, standard lens, and the size and number of subjects. Frames A to B (shot 1) and frames E to F (shot 3) are long shots, two shots (2S), and frames C to D (shot 2) are bust shots and one shot (1S).

  In such a case, since relatively dark images of moonlight night are continuous in Shot 1, if the illumination light is controlled according to the luminance and chromaticity of each frame of these images, the illumination light becomes relatively dark. Then, when shot 1 is switched to shot 2, the ghost photographed by the bust shot becomes a relatively bright image. At this time, if the illumination light is controlled for each frame according to the above-described conventional technique, the illumination light control is largely switched at the time of switching shots, resulting in bright illumination light. When switching to shot 3 again, it returns to the dark illumination light similar to shot 1.

  In other words, even though it is a series of images shot under a certain lighting condition, illumination light that does not take into account the lighting condition at the time of shooting is generated and irradiated, so the atmosphere of the scene is disturbed on the contrary. This will make the viewer feel uncomfortable. Also, if the illumination light becomes darker or brighter in a single segment where a single scene setting (atmosphere) is continuous, the atmosphere of the scene is also hindered and the viewer feels uncomfortable.

  FIG. 39 is a diagram for explaining another example of the problem of the illumination control according to the conventional technique. In the example shown in FIG. 37, a scene of a video shot with a scene setting of outdoors in a sunny day is created. This scene is made up of images obtained by a series of camera work shootings without switching the camera, but the brown dog that is the subject (foreground) is gradually changing from a long jot to an up shot by zoom shooting.

  In other words, the camera work situation used for shooting this scene is that the camera position, angle, number of subjects, camera movement are high positive, high angle, one shot (1S), zoom, subject size is Frames A to B are long shots, frame C is a full figure, frame D is a bust shot, frame E is an up shot, camera lens type is frame A is a standard lens, and frames B and thereafter are telephoto lenses.

  When the illumination light is controlled according to the luminance and chromaticity of each frame of these images, the green light changes from strong illumination light to brown illumination light. In other words, even though the video was shot under constant natural light illumination, illumination light that does not take into account the lighting conditions at the time of shooting is generated and irradiated. Will give a sense of incongruity. In addition, if the color of the illumination light changes in a single segment of scenes in which one scene setting (atmosphere) is continuous, the atmosphere of the scene is still disturbed and the viewer feels uncomfortable.

  FIG. 40 is a diagram for explaining another example of the problem of the illumination control according to the conventional technique. In the example shown in FIG. 38, a scene of a video shot with a scene setting of outdoors in a sunny day is created. This scene consists of images obtained by a series of camera work shooting without switching the camera, but the person wearing pink clothes as the subject (foreground) gradually changes from long jot to bust shot by zoom shooting. is doing.

  In other words, the camera work situation used for shooting this scene is that the camera position, angle, number of subjects, camera movement are eye height, horizontal angle, one shot (1S), zoom, subject size is The frames A to B are long shots, the frame C is a full figure, the frame D is a waist shot, the frame E is an upshot, the camera lens type is a standard lens, and the frames B and thereafter are telephoto lenses.

  When the illumination light is controlled according to the luminance and chromaticity of each frame of these images, the blue light changes from strong illumination light to pink illumination light. In other words, even though the video was shot under constant natural light illumination, illumination light that does not take into account the lighting conditions at the time of shooting is generated and irradiated. Will give a sense of incongruity. In addition, if the color of the illumination light changes in a single segment of scenes in which one scene setting (atmosphere) is continuous, the atmosphere of the scene is still disturbed and the viewer feels uncomfortable.

  FIG. 41 is a diagram for explaining another example of the problem of the illumination control according to the conventional technique. In the example shown in FIG. 41, a scene of an image shot with a scene setting of outdoors in a sunny day is created. This scene consists of images obtained by a series of camera work shooting without switching the camera. By switching the camera lens and shooting, a person wearing pink clothes as the subject (foreground) is a long jot. Has changed from a bust shot to a bust shot.

  In other words, the camera work situation used for shooting this scene is that the camera position, angle, number of subjects, camera movements are eye height, horizontal angle, one shot (1S), fix, subject size is The frames A to C are long shots, the frames D to E are bust shots, and the camera lens type is a standard lens, the frames A to C are telephoto lenses, and the telephoto lenses after the frame D.

  When the illumination light is controlled in accordance with the luminance and chromaticity of each frame of these images, the blue light suddenly changes to pink illumination light. In other words, even though the video was shot under constant natural light illumination, illumination light that does not take into account the lighting conditions at the time of shooting is generated and irradiated. Will give a sense of incongruity. In addition, if the color of the illumination light changes in a single segment of scenes in which one scene setting (atmosphere) is continuous, the atmosphere of the scene is still disturbed and the viewer feels uncomfortable.

  Also, video scenes are usually created by shooting under appropriately set lighting conditions as a segment of video based on a series of scene settings, for example, for the purpose of video producers (screenwriters, directors, etc.) The Therefore, in order to increase the presence and the atmosphere when viewing the video, it is desirable to irradiate the viewing space with illumination light according to the illumination status at the time of scene shooting of the displayed video.

  However, the above-described conventional viewing environment control apparatus detects the feature amount (color signal and luminance signal) for each frame (screen) in the video signal to be displayed, and controls the illumination light. Depending on the situation, it may be difficult to generate illumination light according to the scene situation of the video. For example, if you illuminate the surrounding area with improper color illumination light due to the influence of the clothes of the subject person included in the video signal or artifacts in the background of the subject, the atmosphere of each scene can be reproduced. , Can not maintain the sense of reality for each scene. In other words, viewing environment lighting that deviates significantly from the lighting conditions used at the time of photographing each scene is detrimental to reality.

  In the above-described conventional technology, the illumination light state changes according to the change of the luminance and hue of the video signal for each frame, and particularly when the degree of change in luminance and hue between frames is large. The problem is that the light changes complicatedly and the viewer feels uncomfortable with the flicker. Furthermore, it is not preferable that the illumination light fluctuates in accordance with changes in luminance and hue for each frame during the display of one scene in which the illumination state at the time of shooting does not change, which adversely inhibits the atmosphere for each scene. .

  FIG. 37 is a diagram for explaining an example of the problem of the illumination control according to the conventional technique, and shows a part of continuous moving images. In the example shown in FIG. 37, a scene of a video shot with a scene setting of outdoors in a sunny day is created. This scene is made up of images obtained by shooting through a series of camera work without switching the camera. In this example, an image of a skier sliding down from above the camera toward the vicinity of the camera is taken. Skiers are dressed in red and the sky is clear.

  In other words, this video scene was shot under sunlight (natural light) during sunny days in winter, and this sunlight is a key light. The nature of the illumination light source of this key light is a spotlight (point light source). ), The incident direction of the horizontal illumination light is forward light, the incident direction of the vertical illumination light is top light, the illumination intensity is strong, and the illumination color temperature is about 6000K.

  The video of this scene has a large blue sky area in the initial frame, and the skier's red clothing area gradually increases as the skier slides down and approaches the camera. That is, as the video in the scene progresses, the ratio of the colors constituting each frame changes.

  In such a case, when the illumination light is controlled using the chromaticity and luminance for each frame, the blue light changes from strong illumination light to red illumination light. In other words, even though the video was shot under the lighting conditions described above, illumination light that does not take into account the lighting conditions at the time of shooting is generated / irradiated. Will give a sense of incongruity. In addition, if the color of the illumination light changes in a single segment of scenes in which one scene setting (atmosphere) is continuous, the atmosphere of the scene is still disturbed and the viewer feels uncomfortable.

  FIG. 38 is a diagram for explaining another example of the problem of the illumination control according to the conventional technique. In the example shown in FIG. 38, a scene of an image shot with a scene setting of “moonlit night outdoors” is created. This scene is composed of three shots (1, 2, 3) with different camera work. In Shot 1, the camera takes a long shot of the target ghost. Then, when switching to shot 2, the ghost is shot in an up shot. In shot 3, the camera returns to the camera position of shot 1 again. These shots are intended and configured as a segment of a scene in which one atmosphere continues even though the camera work is different.

  When shooting this scene, the illumination light source is flood light (surface light source), the horizontal illumination light incident direction is forward light, the vertical illumination light incident direction is top light, the illumination intensity is normal, the illumination color Key light with a temperature of about 7000K, illumination light source is flood light (surface light source), horizontal light incident direction is Rembrandt light (left), vertical light incident direction, and the illumination intensity is weak Fill light with an illumination color temperature of about 7000K, the nature of the illumination light source is spotlight (point light source), the incident direction of horizontal light is side light (right), the incident direction of vertical light is forward light, A touch light having a slightly weak illumination intensity and an illumination color temperature of about 5000K is used.

  In such a case, since relatively dark images of moonlight night are continuous in Shot 1, if the illumination light is controlled according to the luminance and chromaticity of each frame of these images, the illumination light becomes relatively dark. When shot 1 is switched to shot 2, the ghost captured in the up shot becomes a relatively bright image. At this time, if the illumination light is controlled for each frame according to the above-described conventional technique, the illumination light control is largely switched at the time of switching shots, resulting in bright illumination light. When switching to shot 3 again, it returns to the dark illumination light similar to shot 1.

  In other words, although it is a series of images shot under the lighting conditions described above, illumination light that does not take into account the lighting conditions at the time of shooting is generated and irradiated, so that the atmosphere of the scene is disturbed on the contrary. This will make the viewer feel uncomfortable. Also, if the illumination light becomes darker or brighter in a single segment where a single scene setting (atmosphere) is continuous, the atmosphere of the scene is also hindered and the viewer feels uncomfortable.

  Furthermore, a video scene is usually created as a segmented video based on a series of scene settings, for example, by the intention of a video producer (screenwriter, director, etc.). Therefore, in order to increase the presence and atmosphere when viewing the video, it is desirable to irradiate the viewing space with illumination light according to the scene status of the display video (scene setting status on the story).

  However, the above-described conventional viewing environment control apparatus detects the feature amount (color signal and luminance signal) for each frame (screen) in the video signal to be displayed, and controls the illumination light. Depending on the situation, it may be difficult to generate illumination light according to the scene situation of the video. For example, if you illuminate the surrounding area with improper color illumination light due to the influence of the clothes of the subject person included in the video signal or artifacts in the background of the subject, the atmosphere of each scene can be reproduced. , Can not maintain the sense of reality for each scene.

  That is, although the illumination at the time of shooting based on the scene setting is generally characterized for each scene, the viewing environment illumination greatly deviating from the feature of this scene is rather detrimental to reality.

  In the above-described conventional technology, the illumination light state changes according to the change of the luminance and hue of the video signal for each frame, and particularly when the degree of change in luminance and hue between frames is large. The problem is that the light changes complicatedly and the viewer feels uncomfortable with the flicker. Furthermore, it is not preferable that the illumination light fluctuates in accordance with changes in luminance and hue for each frame during the display of one scene with no change in scene setting, which adversely inhibits the atmosphere for each scene.

  FIG. 37 is a diagram for explaining an example of the problem of the illumination control according to the conventional technique, and shows a part of continuous moving images. In the example shown in FIG. 37, a scene of a video shot with a scene setting of outdoors in a sunny day is created. This scene is made up of images obtained by shooting through a series of camera work without switching the camera. In this example, an image of a skier sliding down from above the camera toward the vicinity of the camera is taken. Skiers are dressed in red and the sky is clear.

  The video of this scene has a large blue sky background in the initial frame, and the skier's red clothing area gradually increases as the skier slides down and approaches the camera. That is, as the video in the scene progresses, the ratio of the colors constituting each frame changes.

  In such a case, when the illumination light is controlled using the chromaticity and luminance for each frame, the blue light changes from strong illumination light to red illumination light. In other words, even though it is a series of scene situations under sunlight during winter daylight, illumination light that does not take this scene situation into consideration is generated and irradiated, so the atmosphere of the scene is disturbed on the contrary. This will make the viewer feel uncomfortable. In addition, if the color of the illumination light changes in a single segment of scenes in which one scene setting (atmosphere) is continuous, the atmosphere of the scene is still disturbed and the viewer feels uncomfortable.

  FIG. 38 is a diagram for explaining another example of the problem of the illumination control according to the conventional technique. In the example shown in FIG. 38, a scene of an image shot with a scene setting of “moonlit night outdoors” is created. This scene is composed of three shots (1, 2, 3) with different camera work. In Shot 1, the camera takes a long shot of the target ghost. Then, when switching to shot 2, the ghost is shot in an up shot. In shot 3, the camera returns to the camera position of shot 1 again. These shots are intended and configured as a segment of a scene in which one atmosphere continues even though the camera work is different.

  In such a case, in shot 1, a relatively dark image of moonlight night is continuous. When the illumination light is controlled according to the luminance and chromaticity of each frame of these images, the illumination light becomes relatively dark. When shot 1 is switched to shot 2, the ghost captured in the up shot becomes a relatively bright image. At this time, if the illumination light is controlled for each frame according to the above-described conventional technique, the illumination light control is largely switched at the time of switching shots, resulting in bright illumination light. When switching to shot 3 again, it returns to the dark illumination light similar to shot 1.

  In other words, even though it is a series of scene situations under the moonlight in the late night of the historical drama, illumination light that does not take this scene situation into consideration is generated and irradiated, so the atmosphere of the scene is disturbed and viewers are disturbed It will give you a sense of incongruity. Also, if the illumination light becomes darker or brighter in a single segment where a single scene setting (atmosphere) is continuous, the atmosphere of the scene is also hindered and the viewer feels uncomfortable.

  The present invention has been made in view of the above-described problems, and is a data transmission device, a data transmission method, a viewing environment control device, which can realize optimal viewing environment lighting control according to the content of a display video, An object is to provide a viewing environment control system and a viewing environment control method.

  According to a first aspect of the present invention, in a data transmitting apparatus that transmits video data composed of one or more frames, the video data is divided into segments composed of an arbitrary number of one or more frames. Adding segment delimiter information indicating a start time and / or section and at least one of camera work information, shooting illumination information, and scene setting information at the time of shooting of the video data to the video data in units of segments. Features.

  The second invention of the present application is characterized in that the camera work information includes at least information representing a camera position at the time of photographing each segment.

  The third invention of the present application is characterized in that the camera work information includes at least information representing a camera angle at the time of photographing each segment.

  The fourth invention of the present application is characterized in that the camera work information includes at least information indicating a size of a subject at the time of photographing each segment.

  The fifth invention of the present application is characterized in that the camera work information includes at least information indicating the number of subjects at the time of photographing each segment.

  The sixth invention of the present application is characterized in that the camera work information includes at least information indicating the movement of the camera at the time of photographing each segment.

  The seventh invention of the present application is characterized in that the camera work information includes at least information indicating a type of a camera lens used for photographing each segment.

  The eighth invention of the present application is characterized in that the photographing illumination information includes at least information indicating a lighting type of illumination used for photographing each segment.

  The ninth invention of the present application is characterized in that the photographing illumination information includes at least information indicating the nature of the illumination used for photographing each segment.

  The tenth invention of the present application is characterized in that the photographing illumination information includes at least information indicating the direction of illumination light used for photographing each segment.

  The eleventh invention of the present application is characterized in that the photographic illumination information includes at least information indicating the intensity of illumination light used for photographing each segment.

  In a twelfth aspect of the present invention, the photographing illumination information includes at least information indicating the color temperature of the illumination light used for photographing each segment.

  The thirteenth invention of the present application is characterized in that the scene setting information includes at least information representing a season on a story of each segment.

  A fourteenth invention of the present application is characterized in that the scene setting information includes at least information indicating a time on a story of each segment.

  The fifteenth aspect of the present invention is characterized in that the scene setting information includes at least information indicating a place on the story of each scene.

  The sixteenth invention of the present application is characterized in that the scene setting information includes at least information representing an age on a story of each segment.

  The seventeenth invention of the present application is characterized in that the scene setting information includes at least information indicating the weather on the story of each segment.

  According to an eighteenth aspect of the present invention, the scene setting information includes at least information representing a region on a story of each segment.

  According to a nineteenth aspect of the present application, in response to an external request, segment delimiter information indicating the start time and / or section of a segment obtained by dividing video data into one or more arbitrary numbers of frames, and imaging of the video data At least one of camera work information, photographing illumination information, and scene setting information at the time is transmitted in the segment unit.

  According to a twentieth aspect of the present invention, video data to be displayed on a display device, segment delimiter information indicating a start time and / or a section of a segment obtained by dividing the video data into one or more arbitrary numbers of frames, and the segment Receiving means for receiving at least one of camera work information, shooting illumination information, and scene setting information at the time of shooting of the video data, which is associated with the video data in units; the segment delimiter information; and the video Control means for controlling the illumination light of the illumination device installed around the display device based on at least one of data and / or camera work information, photographing illumination information, and scene setting information. Features.

  A twenty-first invention of the present application is characterized in that the control means switches and controls the illumination light of the illumination device in units of segments.

  According to a twenty-second aspect of the present application, a video reference mode for controlling illumination light of the lighting device using at least a feature amount of the video data, and a feature amount of the video data are used. And an additional information reference mode for controlling illumination light of the illumination device by using at least one of the photographic illumination information and scene setting information without using it.

  A twenty-third invention of the present application is characterized in that, when the video reference mode is set, the control means limits a screen area for detecting a feature amount of the video data according to the camera work information. To do.

  In a twenty-fourth aspect of the present application, when the video reference mode is set, the control means includes at least an estimated illumination value obtained based on a feature amount of the video data, the photographic illumination information, and the scene setting information. A viewing environment control apparatus that controls illumination light of the illumination apparatus using an estimated illumination value obtained based on any of the above.

  A twenty-fifth aspect of the present invention includes the viewing environment control device and a lighting device whose viewing environment illumination light is controlled by the viewing environment control device.

  According to a twenty-sixth aspect of the present invention, in a data transmission method for transmitting video data composed of one or more frames, the video data is divided into segments composed of an arbitrary number of one or more frames. Segment segment information indicating a start time and / or section and at least one of camera work information, shooting illumination information, and scene setting information at the time of shooting the video data are added to the video data in units of segments. It is characterized by transmitting.

  According to a twenty-seventh aspect of the present application, in response to an external request, segment delimiter information indicating the start time and / or section of a segment obtained by dividing video data into one or more arbitrary numbers of frames, and imaging of the video data At least one of camera work information, photographing illumination information, and scene setting information at the time is transmitted in the segment unit.

  According to a twenty-eighth aspect of the present invention, video data to be displayed on a display device, segment delimiter information indicating a start time and / or a section of a segment obtained by dividing the video data by one or more arbitrary frames, and the segment Receiving at least one of camera work information, shooting illumination information, and scene setting information at the time of shooting of the video data, which is associated with the video data in units, and receives the segment delimiter information, the video data, and / or Or the illumination light of the illuminating device installed around the said display apparatus is controlled based on at least any one of camera work information, imaging | photography illumination information, and scene setting information.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to implement | achieve the optimal viewing-and-listening environment according to the place (atmosphere) at the time of imaging | photography of an image | video.

<First Embodiment>
Hereinafter, a first embodiment of the viewing environment control system of the present invention will be described in detail with reference to FIGS. Here, FIG. 1 is a block diagram showing a schematic configuration of a main part of a video transmission apparatus in the viewing environment control system of the present embodiment, and FIG. 2 shows an example of an output bit stream of the video transmission apparatus in the viewing environment control system of the present embodiment. FIG. 3 is a diagram for explaining an example of the data structure of additional data in the viewing environment control system of the present embodiment, FIG. 4 is a diagram for explaining the components of the video, and FIG. 5 is the present embodiment. It is explanatory drawing which shows an example of the scene delimiter information in the viewing-and-listening environment control system.

  6 is an explanatory diagram showing an example of camera work information in the viewing environment control system of the present embodiment, FIG. 7 is a diagram for explaining the camera position and camera angle, and FIG. 8 is a diagram for explaining the subject size. 9 is a diagram for explaining the number of subjects, FIG. 10 is an explanatory diagram showing an example of photographing illumination information in the viewing environment control system of the present embodiment, and FIG. 11 is a diagram for explaining division of photographing illumination light in the horizontal direction and the vertical direction. FIG. 12 is an explanatory diagram showing an example of scene setting information in the viewing environment control system of the present embodiment.

  FIG. 13 is a block diagram showing a schematic configuration of the main part of the video receiving apparatus in the viewing environment control system of this embodiment, FIG. 14 is a block diagram showing the configuration of the illumination control data generation unit in FIG. 13, and FIG. FIG. 16 is a flowchart showing the illumination estimation processing in the video reference mode in the viewing environment control system of this embodiment, and FIG. 17 is the viewing environment control of this embodiment. It is a flowchart which shows the illumination estimation process at the time of the additional information reference mode in a system.

  FIG. 18 is a flowchart showing illumination estimation processing using camera work information in the viewing environment control system of this embodiment, and FIG. 19 is a place (atmosphere) estimation target determined by the camera position and camera angle in the viewing environment control system of this embodiment. FIG. 20 is an explanatory diagram showing an example of a field (atmosphere) estimation target area determined by the camera position in the viewing environment control system of the present embodiment, and FIG. 21 is a camera angle in the viewing environment control system of the present embodiment. It is explanatory drawing which shows the example of a field (atmosphere) estimation object area | region decided by.

  FIG. 22 is an explanatory view showing an example of a field (atmosphere) estimation target area determined by the subject size and the number of subjects in the viewing environment control system of the present embodiment, and FIG. 23 is a field (depending on the subject size in the viewing environment control system of the present embodiment ( FIG. 24 is an explanatory diagram illustrating an example of a field (atmosphere) estimation target area determined by the number of subjects in the viewing environment control system of the present embodiment. FIG. 25 is a viewing environment control of the present embodiment. It is a figure which shows the screen feature-value detection area | region in a system.

  Further, FIG. 26 is a flowchart showing illumination estimation processing using the photographic illumination information in the viewing environment control system of the present embodiment, FIG. 27 is an explanatory diagram showing an example of control data in the viewing environment control system of the present embodiment, and FIG. Is an explanatory diagram showing an example of the relationship between the illumination direction information and the corresponding illumination device ID in the viewing environment control system of the present embodiment, and FIG. 29 is an explanatory diagram showing an example of the photographic illumination information corresponding to the video scene shown in FIG. 30 is an explanatory diagram showing the color reproduction range of illumination light in the viewing environment control system of the present embodiment, and FIG. 31 is an explanatory diagram showing photographing illumination information corresponding to the video scene shown in FIG.

  FIG. 32 is a flowchart showing illumination estimation processing using scene setting information in the viewing environment control system of this embodiment, and FIG. 33 is an explanation showing an example of the relationship between scene setting information and control data in the viewing environment control system of this embodiment. FIG. 34 and FIG. 34 are other explanatory views showing the color reproduction range of illumination light in the viewing environment control system of the present embodiment.

  As shown in FIG. 1, the video transmission apparatus (data transmission apparatus) in this embodiment includes segment data supplied as video data, audio data, and additional data, scene setting information, photographing illumination information, and camera work information. A data multiplexer 1 that divides and multiplexes at least one of them into a transport stream packet (TSP) format, and an error correction code is added to the output data of the data multiplexer 1. And a transmission unit 2 that performs modulation and sends the data as broadcast data to a transmission line.

  FIG. 2 is an explanatory diagram showing an outline of the structure of a transport stream packet (TSP) defined by MPEG2 (Moving Picture Experts Group 2) -Systems, and 11 is the data content of TSP and other data defined by MPEG2-Systems. A header in which information is described, 12 is an extension header (adaptation field) that can describe information determined by the sender, and 13 is a payload composed of data such as video and audio.

  In the present embodiment, for example, video data and audio data can be transmitted using the payload 13, and additional data can be transmitted using the extension header (adaptation field) 12. Note that different data streams of video data, audio data, and additional data may be multiplexed and transmitted. Further, the additional data may be described in a text format, or may be encoded and described in a binary format.

  FIG. 3 is a diagram for explaining an example of the data structure of the additional data. In the present embodiment, video data is divided into segments each composed of an arbitrary number of one or more frames, and various additional information related to each segment is described in segment units.

  Here, the segment is defined as a configuration in which video content is divided by an arbitrary length in the time direction. For example, the segment is a segment corresponding to a scene, a shot, a frame, or the like. For example, when a segment corresponds to a scene, information existing in shot units is collectively described in the segment. When a segment corresponds to a shot, the same information is repeatedly described in all the corresponding segments as information existing in a scene unit. In addition, when a segment corresponds to a frame, the same information is repeatedly described in all the corresponding segments as information existing in scene units and shot units.

  Note that the viewing environment illumination is mainly controlled in units of scenes in which the situation of the scene is constant, but is often controlled in units of shots in which the situation of shooting is constant. In some cases, fine-grained control in units of frames is sometimes desirable, so the amount of data to be added can be reduced by dividing the segments into units that often require the viewing environment lighting to be controlled to be constant depending on the content of the display data. The additional data can be described efficiently while suppressing the increase.

  Next, the configuration of a video including scenes and shots will be described with reference to FIG. Video data constituting a continuous moving image can be considered in a three-layer structure as shown in FIG. The 1st layer which comprises an image | video (Video) is a flame | frame (Frame). A frame is a physical layer and refers to a single two-dimensional image. Frames are usually obtained at a rate of 30 frames per second. The second layer is a shot. A shot is a sequence of frames taken by a single camera. The third layer is a scene. A scene is a sequence of shots having a story-like connection.

  As shown in FIG. 5, the segment delimiter information is composed of a segment number, a start time, and section information. The segment start time is specified by a time code added to indicate playback time information of each of the video data and audio data. For example, as shown in FIG. 5, the time (h): minute (m) of the video data : Second (s): It is composed of information indicating a frame (f). The segment section is composed of information indicating a time (h): minute (m): second (s): frame (f) indicating a period from the time indicating the start time code of the segment to the start time code of the next segment. Has been.

  Next, camera work information indicating the camera work status at the time of shooting video data will be described in detail with reference to FIG. In this embodiment, as camera work information, position information indicating the camera position at the time of video shooting, angle information indicating the camera angle at the time of video shooting, subject size information indicating the size of the subject at the time of video shooting, subject at the time of video shooting Information on the number of subjects, camera motion information representing the movement of the camera during video shooting, and lens type information representing the type of camera lens used for video shooting. All of these pieces of information are useful information for estimating the illumination environment at the time of shooting video data and producing the atmosphere and presence of each segment with illumination light. Hereinafter, each information is demonstrated.

  First, camera positions at the time of video shooting are classified into high position (high positive), eye height, and low position (low positive). As shown in FIG. 7A, the high positive is a position that is often used for a long shot, which will be described later, and is a high position that expresses a vast expanse or is used for crossing a crowd. The eye height is a normal position and the eye height position of the subject. The low positive is a low position such as a child's eyes.

  In the case of outdoor shooting, the height of the horizontal line (horizon line) in the shooting screen tends to be low at the high positive position and high at the low positive position. Therefore, the video feature amount is detected according to the camera position at the time of video shooting. By limiting the screen area and controlling the viewing environment illumination described later, the realistic sensation of the display video scene can be improved. Therefore, in the present embodiment, as the position information, a 2-bit description indicating whether the camera position at the time of video shooting of each segment belongs to high positive / eye height / low positive is included in the camera work information. Yes.

  Further, the camera angle at the time of video shooting is classified into a high angle (overhead view), a horizontal angle (line of sight), and a low angle (tilt). As shown in FIG. 7B, the high angle is a method of photographing at an angle looking down, and is used for an overall objective situation description. The horizontal angle is a natural and standard angle in which the camera is directed horizontally. Low angle is a method of shooting at an angle looking upward, and is used to express intimidation, superiority, victory, and the like.

  For outdoor shooting, the horizontal line (horizon) in the shooting screen tends to be high at high angles and low at low angles. The presence of the display video scene can be improved by limiting the screen area to be detected and controlling the viewing environment illumination described later. Therefore, in this embodiment, as the angle information, a 2-bit description indicating whether the camera angle at the time of video shooting of each segment belongs to high angle (overhead view), horizontal angle (line of sight), or low angle (tilt). Is included in the camera work information.

  Furthermore, the subject size (also referred to as screen size) at the time of video shooting is classified into a long shot, a full figure, a knee shot, a waist shot, a bust shot, an up shot, a close-up shot, and a big close-up shot. As shown in FIG. 8, a long shot (LS) is a studio landscape or an outdoor wide-angle shot, and is often used for the overall positional relationship or the beginning and end of a story. The full figure (F.F.) has a size in which the whole body of the subject (a whole body image from the toes to the head when standing, a whole body image when sitting) is included in the shooting screen. The knee shot (K.S.) is a size in which the subject is placed from above the knee to the head in the shooting screen.

  Waist shot (WS) is the size of a half-body image with the subject's waist or hips up to the head in the shooting screen, and is used for basic news and talker shots, making it easy to insert telops. It is. The bust shot (B.S.) is the size of the upper body with the subject's chest or the top of the chest up to the head in the shooting screen, and this is the size most often used for human images. The up shot (US) is a shot that expresses the expression and excitement of the subject. The close-up shot (C.U.) is a shot that emphasizes up, and has a size that can cut the head (hair) of the subject. A big close-up shot (B.C.U.) is used to emphasize close-up.

  In this way, the area of the background changes depending on the subject size (screen size) at the time of video shooting, so the segments for detecting video feature values can be limited according to the subject size (screen size) at the time of video shooting. By limiting the screen area for detecting the video feature amount and controlling the viewing environment illumination described later, it is possible to improve the realism of the display video scene. Therefore, in the present embodiment, as subject size information, the subject size at the time of video shooting of each segment is long shot / full figure / knee shot / waist shot / bust shot / up shot / close up shot / big close up shot. Is included in the camera work information.

  Further, the number of subjects at the time of video shooting is classified into only one shot (1S), two shot (2S), three shot (3S), group shot (GS), and landscape (background). As shown in FIG. 9, one shot has only one (1) foreground subject, two shots have two (2) foreground subjects, and three shots have three foreground subjects (three). ), A group shot is a shot when there are four or more subjects as foregrounds.

  In this way, the area of the background changes depending on the number of subjects at the time of video shooting. Therefore, the screen area for detecting video feature values is limited to control the viewing environment illumination described later according to the number of subjects at the time of video shooting. By doing so, the presence of the displayed video scene can be improved. Therefore, in this embodiment, as the number-of-subjects information, the number of subjects at the time of video shooting of each segment is one shot (1S) / two shot (2S) / three shot (3S) / group shot (GS) / landscape (background). ) Only includes a 3-bit description indicating that it belongs to the camera work information.

  Furthermore, the movement of the camera at the time of video recording is classified into fix, pan, tilt (tilt), roll, zoom, dolly, and follow. Fix is a shooting method that does not move the camera position without zooming to change the angle of view. Pan is a shooting method that shows the surroundings by shaking the camera in the horizontal direction, and is used when it is desired to explain the situation or show the positional relationship between the left and right objects. Tilt is a shooting method that shakes the camera vertically, such as shooting from the bottom to the top (from top to bottom) or swinging up along the trunk of a tree to shoot branches and leaves. Sometimes used.

  The roll is a photographing method for rotating the camera, and the zoom is a photographing method for making a large image or a wide size with a zoom lens. Dolly is a technique for shooting while the camera itself moves, and follow is a technique for shooting while following the movement of the subject when shooting a moving subject such as a running person or a vehicle.

  As described above, the area of the background and the feature amount change depending on the movement of the camera at the time of video shooting. Therefore, the segment for detecting the video feature amount is limited according to the movement of the camera at the time of video shooting, and the viewing described later. By controlling the environmental lighting, it is possible to improve the presence of the displayed video scene. Therefore, in this embodiment, as the camera motion information, 3 bits indicating whether the camera motion at the time of video shooting of each segment belongs to fix / pan / tilt (tilt) / roll / zoom / dolly / follow. Is included in the camera work information.

  Further, the lens work used at the time of video shooting is classified into a standard lens, a wide-angle lens, a telephoto lens, and a macro lens. The standard lens is a lens that is close to the human visual field and capable of capturing a natural perspective. The wide-angle lens has a wide angle of view and can shoot a wide range, and is often used for taking a landscape or a group photo. A telephoto lens is a lens that can shoot a large subject by attracting a far-off subject, and is often used for one scene of a sport that cannot be actually approached, portrait photography, and the like. The macro lens is a close-up lens that is suitable for close-up photography of flowers and insects.

  In this way, the area of the background changes depending on the lens work at the time of video shooting, so the segment for detecting video feature values is limited to control the viewing environment illumination described later, adapting to the movement of the camera at the time of video shooting By doing so, the presence of the displayed video scene can be improved. Therefore, in the present embodiment, as the lens type information, a 2-bit description indicating whether the type of the camera lens used at the time of shooting the image of each segment belongs to a standard lens / a wide-angle lens / a telephoto lens / a macro lens. , Contained in camera work information.

  It should be noted that various information included in the camera work information is not limited to the above-described information, and it is obvious that more detailed information may be added. Further, the camera work information described above is created based on a script (also called a scenario or a script) and can be generated using a camera work program or the like used in a video shooting site. It becomes possible to omit the work of creating information. Further, since this camera work information often changes in units of shots, the camera work information can be efficiently described in units of segments by dividing the segments in accordance with the shots.

  Next, photographing illumination information indicating a lighting state at the time of photographing video data will be described in detail with reference to FIG. In the present embodiment, as shooting illumination information, lighting type information indicating the lighting type of the illumination used at the time of video shooting, lighting property information indicating the characteristics of the illumination light source used at the time of video shooting, and the illumination used at the time of video shooting Video shooting of illumination light direction (horizontal direction) information representing the incident direction (horizontal direction) of light, and illumination light direction (vertical direction) information representing the incident direction (vertical direction) of the illumination light used during video shooting Illumination intensity information representing the intensity of the illumination light used at times, and illumination color temperature information representing the color temperature of the illumination light used at the time of video shooting. All of these pieces of information are useful information for estimating the illumination environment at the time of shooting video data and producing the atmosphere and presence of each segment with illumination light. Hereinafter, each information is demonstrated.

  First, keylights, filllights, touchlights, and baselights are known as commonly used lighting types for photographing illumination. The key light is generated by the main light source, and is light on the setting of the scene, which is a central light. For example, in a location outside the room during the daytime, the key light corresponds to ordinary sunlight. Fill light is an auxiliary light source that erases shadows and brightens or shines the subject. For example, it reflects sunlight by a reflector at an outdoor location in the daytime and is dark. It is this light that hits the place.

  The touch light is light for separating the subject from the background or giving a three-dimensional effect, and is light that is effective by being irradiated obliquely from behind to express darkness. Base light is light used to brighten the entire image on average or to relieve strong contrast.

  Usually, shooting is performed using a combination of one or more lightings. Therefore, the realism of the display video scene is improved by controlling the viewing environment lighting described later in accordance with the lighting conditions for each lighting type. be able to. Therefore, in the present embodiment, as the lighting type information, a 2-bit description indicating whether the illumination light at the time of video shooting of each segment belongs to key light / fill light (holding) / touch light / base light, It is included in the shooting lighting information.

  Moreover, the property of the illumination light source used for each lighting type is determined. Typical properties of an illumination light source at the time of photographing include a spotlight (point light source) and a floodlight (surface light source). A spotlight (point light source) is illumination in which the directivity of light is clear, and is an illumination in which shadows appear clearly. On the other hand, floodlight (surface light source) is illumination used to obtain uniform brightness over the entire space, and is a illumination in which shadows are blurred and shades do not appear strongly.

  Usually, since the nature of the illumination light source is determined for each lighting type, the realism of the display video scene can be improved by controlling the viewing environment illumination described later in accordance with the nature of the illumination light source. . For example, when reproducing illumination light by a spotlight (point light source), the illumination range by the viewing environment illumination device is narrowed, and when reproducing illumination light by a floodlight (surface light source), the illumination range by the viewing environment illumination device Can be widened.

  Therefore, in the present embodiment, as the illumination property information, 1-bit indicating whether the property of the illumination light source used at the time of video shooting of each segment belongs to spotlight (point light source) or flood light (surface light source). A description is included in the photographic lighting information.

  Furthermore, since the direction of the illumination light is determined for each lighting type, the realism of the display video scene can be improved by controlling the viewing environment illumination described later in accordance with the direction of the illumination light. In other words, by independently controlling a plurality of lighting devices arranged on the top, bottom, left, and right of the video display device, it is possible to finely reproduce the shooting lighting environment during video shooting.

  Here, as shown in FIG. 11A, the illumination light is divided into the following directions: forward light, side light (left, right), backlight, rim light (left, right), Rembrandt light (left, right). It is divided into. Forward light is light from the front where brightness can be obtained most efficiently. The side light is a light that can enhance the three-dimensional effect, but has a strong shadow and a strong impression with a strong contrast. Backlight is light from the back that can create a silhouette effect that raises the outline and produces a fantastic effect. Rimlight is light in a semi-backlit direction used for emphasizing three-dimensional effects and producing hair details. Rembrandt light is a 45-degree light that best expresses facial features and makes them look beautiful.

  Further, as shown in FIG. 11B, the illumination light is divided into a normal light, a back light, a top light, a tilt light, and a Rembrandt light. Forward light is light from the front where brightness can be obtained most efficiently. Backlight is light from the back that can create a silhouette effect that raises the outline and produces a fantastic effect. The top light is light from almost right above the subject that is used for characteristic image expression rather than showing a person beautifully. Aori is light from the lower front of the subject used for the purpose of special effects. Rembrandt light is a 45-degree light that best expresses facial features and makes them look beautiful.

  Therefore, in this embodiment, as the illumination light direction (horizontal direction) information, the incident direction (horizontal direction) of the illumination light at the time of shooting each video scene is the following light / side light (left) / side light (right). In the photographing illumination information, there is included a 3-bit description indicating whether it belongs to: / backlight / rim light (left) / rim light (right) / Rembrandt light (left) / Rembrandt light (right). In addition, as illumination light direction (vertical direction) information, it indicates whether the incident direction (vertical direction) of illumination light at the time of video shooting of each segment belongs to the following light / backlight / top light / tilt / rembrandt light. A 3-bit description is included in the photographic lighting information.

  In addition, since the intensity and color temperature of the illumination light are determined for each lighting type, by controlling the viewing environment illumination, which will be described later, in accordance with the intensity and color temperature of the illumination light, the realism of the display video scene is increased. Can be improved. Here, as the illumination intensity information, 3 bits indicating whether the intensity of the illumination light at the time of shooting each video scene belongs to very weak / weak / somewhat weak / normal / somewhat strong / strong / very strong Is included in the photographic lighting information. As the illumination color temperature information, the color temperature (K) of the illumination light at the time of shooting each video scene is 1000/2000/3000/4000/5000/6000/7000/8000/9000/10000/11000/12000 / The photographic lighting information includes a 4-bit description indicating which of 13000/14000/15000/25000 belongs.

  Needless to say, various types of information included in the photographic illumination information are not limited to those described above. For example, instead of illumination color temperature information directly representing the color temperature of the photographic illumination light, information on the type of lighting fixture (sodium lamp / mercury lamp / LED / fluorescent lamp / candle /.../ metal halide lamp, etc.) and color conversion filter (conversion) Filter) type information and the like may be described, and the color temperature of the illumination light at the time of photographing may be obtained from this.

  In addition, the above-described shooting illumination information can be generated based on a script and can be generated using a lighting plan or the like used in a video shooting site, and in this case, the work of creating new shooting illumination information can be omitted. Is possible. In addition, since the photographing illumination information often changes in shot units or scene units, the photographing illumination information can be efficiently described in segment units by dividing the segments according to shots and scenes.

  Furthermore, scene setting information indicating the scene setting status of each scene of the video data will be described in detail with reference to FIG. In the present embodiment, as the scene setting information, the season information indicating the season on the story of each scene, the time information indicating the time on the story of each scene, the location information indicating the place on the story of each scene, the story of each scene It includes period information representing the above period, weather information representing the weather on the story of each scene, and area information representing the area on the story of each scene. Each of these pieces of information can be used to estimate a characteristic lighting environment related to the scene setting of each scene, and is useful information for directing the atmosphere and presence of each scene with illumination light. .

  First, because the intensity and color temperature of natural light (sunlight) change depending on the season, the realism of outdoor scenes can be improved by controlling the viewing environment lighting described later in accordance with the season set in the story. Can do. That is, for example, the color temperature of sunlight during spring to autumn is about 6000K, but even in the daytime in winter, it is possible to express cold by raising the color temperature of sunlight to about 7000K. . Moreover, although the intensity of sunlight in summer is strong, it is a little stronger in spring and autumn and normal in winter. Therefore, in this embodiment, as the seasonal information, a 2-bit description indicating whether the scene setting of the video scene of each segment belongs to spring / summer / autumn / winter is included in the scene setting information. .

  In addition, since the intensity and color temperature of natural light (sunlight) also change depending on the time of day, by adjusting the viewing environment lighting described later in accordance with the time set on the story, The presence can be improved. That is, for example, the color temperature of sunlight in the early morning is about 2000K, about 4000K in the morning, about 5000K in the morning, about 6000K in the noon, about 5000K in the afternoon, about 3000K in the evening, about 4000K at dusk, about 7000K at night is there. Also, although the intensity of sunlight in the early morning is somewhat weak, it is normal in the morning, slightly strong in the morning, strong in the noon, slightly strong in the afternoon, normal in the evening, slightly weak in the dusk, and very weak in the night. Therefore, in this embodiment, the time information includes a 3-bit description indicating whether the scene setting of the video scene of each segment belongs to early morning / morning / am / noon / afternoon / evening / dusk / night. It is contained in the setting information.

  In addition, since the intensity and color temperature of natural light and artificial light change depending on the location, the realism of indoor and outdoor scenes can be improved by controlling the viewing environment lighting described later in accordance with the location set in the story. be able to. That is, for example, generally at night, there is only light of a slightly weak intensity outdoors due to moonlight or street lighting, but there is a slightly high intensity of light indoors because the interior lighting is lit even at night. In addition, the intensity of the illumination light in the studio is strong day and night, and the intensity of the light becomes normal if it is shaded even outdoors in the daytime. Therefore, in the present embodiment, as the location information, a 2-bit description indicating whether the location setting of the video scene of each segment belongs to indoor / studio / outdoor / outdoor shade is included in the scene setting information. Yes.

  Furthermore, since the intensity and color temperature of natural light and artificial light change depending on the times, the realism of each scene of contemporary and historical plays can be adjusted by controlling the viewing environment lighting described below in accordance with the times set in the story. Can be improved. That is, for example, the color temperature of light of a lighting device generally used in a modern drama is about 5000K for a fluorescent lamp (daylight), about 6700K for a fluorescent lamp (daylight color), and about 2800K for an incandescent lamp. On the other hand, the color temperature of the candle flame often used as a light source at night in historical drama is about 1800-2500K. In addition, lighting intensity tends to be high in modern plays and low lighting intensity in historical plays. Therefore, in the present embodiment, as the period information, the scene setting information includes a 1-bit description indicating whether the period setting of the video scene of each segment belongs to modern / period drama.

  In addition, since the intensity and color temperature of natural light (sunlight) change depending on the weather, the realism of the outdoor scene is improved by controlling the viewing environment lighting described later in accordance with the weather set in the story. be able to. That is, for example, the color temperature of sunny daylight is about 6000K, the daylight in the rainy day is 8000 to 9000K, and the daylight in the cloudy day is about 6500 to 8000K. In addition, the intensity of natural light is strongest when it is clear, and decreases in the order of snow (clear day), clear, cloudy, fog, haze, haze, thunder, and rain. Therefore, in this embodiment, as weather information, it indicates whether the weather setting of the video scene of each segment belongs to clear / sunny / cloudy / fog / haze / haze / thunder / rain / snow (sunny day). A 3-bit description is included in the scene setting information.

  And since the intensity and color temperature of natural light (sunlight) change depending on the region, the realism of the outdoor scene is improved by controlling the viewing environment lighting described later in accordance with the region set in the story. Can be made. That is, for example, the color temperature of sunlight in the tropics (daytime) is about 5000K, about 7000K in the cold zone (daytime), and about 6000K in the dry zone, the temperate zone, and the subarctic zone (daytime). In addition, the intensity of sunlight is very strong in the tropics, and tends to decrease in the order of dry zone, temperate zone, subarctic zone, and cold zone. Therefore, in the present embodiment, as the area information, a 3-bit description indicating whether the area setting of the video scene of each segment belongs to one of the tropical / dry zone / temperate zone / subarctic zone / cold zone is included in the scene setting information. Contains.

  Needless to say, various types of information included in the scene setting information are not limited to those described above. For example, the season information and time information may be described in the scene setting information as date and time information representing the date and time set on the story of each scene, and the region information may be The latitude and longitude information representing the latitude and longitude of the area set on the story may be described in the scene setting information.

  Furthermore, the various information in the scene setting information described above also serves as scene setting status information in scenario information included in broadcast data as program additional information, scene setting status information prepared for editing / searching, and the like. Can do. Further, since this scene setting information changes in units of scenes, the scene setting information can be efficiently described in units of segments by dividing the segments in accordance with the scenes.

  Next, a video reception device (data reception device) that receives broadcast data transmitted from the video transmission device, displays and reproduces video and audio, and controls the viewing environment illumination at that time will be described.

  As shown in FIG. 13, the video receiving apparatus according to the present embodiment receives and demodulates broadcast data input from a transmission path, performs error correction, and outputs video from the output data of the receiving unit 21. A data separation unit 22 for separating / extracting each of video data, TC (time code) output to the display device 25, audio data, TC (time code) output to the audio reproduction device 26, and additional data, and a data separation unit 22 The illumination control data generator 24 outputs the illumination control data (RGB data) obtained based on the additional data and / or the feature amount of the video data / audio data separated to the illumination device 27 that illuminates the viewing environment space. And delay generators 28 and 29 for delaying and outputting video data and audio data by the processing time in the illumination control data generator 24. There.

  The time code is information added to indicate the reproduction time information of each of the video data and the audio data. For example, the time (h): minute (m): second (s): frame (f) of the video data ).

  The illumination device 27 can be configured by an LED that is installed around the video display device 25 and emits light of, for example, three primary colors of RGB having a predetermined hue. However, the illumination device 27 may be configured to be able to control the illumination color and brightness of the surrounding environment of the video display device 25, and is not limited to the combination of LEDs that emit a predetermined color as described above, and is a white LED. And a color filter, or a combination of a white light bulb, a fluorescent tube and a color filter, a color lamp, or the like can be applied. Further, it is sufficient that one or more lighting devices 27 are installed.

  As illustrated in FIG. 14, the illumination control data generation unit 24 of the present embodiment includes an additional data analysis unit 351 that analyzes additional data separated by the data separation unit 22, and a segment extracted by the additional data analysis unit 351. Lighting control corresponding to eight vertices in the virtual viewing environment space based on the delimiter information and the video data, TC, audio data, TC, or scene setting information and / or shooting lighting information and / or camera work information Lighting control data estimation unit 352 that calculates data estimation values (R, G, B), and lighting control data that controls the lighting control data estimation values for one or more lighting devices 27 installed in an actual viewing environment space And an illumination control unit 354 that converts and outputs (R, G, B).

In this embodiment, as shown in FIG. 15, each vertex V _{1 to} V ₈ of a virtual viewing environment space including a video display device (for example, a space consisting of 4500 mm × 3600 mm × 2400 mm which is a standard room size). Each of which can be independently controlled and is provided with eight lighting devices to which unique IDs 1 to 8 are assigned, and based on video feature amount / audio feature amount and / or additional data, Suppose that the illumination control data estimated value (R, G, B) corresponding to each of the eight illumination devices ID1 to 8 is obtained. Here, the origin of the coordinate axes representing the virtual viewing environment space is the screen center of the video display device, and the xz plane is a plane parallel to the screen of the video display device.

  In addition, each plane constituting the virtual viewing environment space is equally divided into four, and an area composed of three planes sharing the same vertex is defined as an illumination unit (irradiation range). The eight illumination devices ID1 to 8 each irradiate eight corresponding illumination units (irradiation ranges), and accordingly, illumination control data estimated values (brightness and color of illumination light that should irradiate each illumination unit) ( R, G, B) is obtained for each of the eight lighting devices ID1 to 8. In FIG. 15, as an example, the illumination unit (irradiation area) corresponding to the illumination device ID 6 is shaded.

  Next, the space environment (the size of the room, the shape, the number of lighting devices, and the position) for actual viewing varies from user to user and is not the same as the virtual viewing environment space described above. It is necessary to obtain the illumination control data (R, G, B) to be output to the illumination device 27 installed in the viewing environment space by performing mapping processing from the virtual viewing environment space to the actual viewing environment space.

  Information indicating the actual viewing environment is given to the lighting control unit 354, and using this information, the lighting control data estimated values (R) corresponding to the eight lighting devices ID1 to 8 in the virtual viewing environment space are used. , G, B), the lighting control data (R, G, B) to be output to the lighting device 27 installed in the actual viewing environment space is calculated.

  Note that the information indicating the actual viewing environment may include at least information indicating the relative position of the illumination device 27 with respect to the video display device 25, either directly or indirectly. The information indicating the relative position of the illumination device 27 may be input, or the information indicating the relative position of the illumination device 27 may be automatically detected using a sensor or a camera.

  Here, an example of a method for calculating illumination control data to be output to the illumination device 27 installed in the actual viewing environment space will be described. First, after converting both viewing environment spaces into the same coordinate system, it is determined whether or not their sizes match. When both viewing environment spaces have the same size, the illumination range by the lighting device 27 installed in the actual viewing environment space and the lighting units corresponding to the eight lighting devices ID1 to 8 in the virtual viewing environment space The overlap with (illumination range) is determined.

  When the illumination range by the illumination device 27 installed in the actual viewing environment space is included in the illumination unit (illumination range) corresponding to any of the illumination devices ID1 to 8 in the virtual viewing environment space, the illumination device 27 The illumination control data for is an illumination control data estimated value for any one of the illumination devices ID1 to 8 corresponding to the illumination unit including the illumination range by the illumination device 27.

  For example, when the illumination range by the illumination device 27 installed in the actual viewing environment space is included in the illumination unit (illumination range) corresponding to the illumination device ID 6 in the virtual viewing environment space, the illumination control for the illumination device 27 is performed. The data is the illumination control data estimated value obtained for the illumination device ID 6 in the virtual viewing environment space.

  On the other hand, when the illumination range by the illumination device 27 installed in the actual viewing environment space spans the illumination units (illumination ranges) corresponding to the plurality of illumination devices ID1 to 8 in the virtual viewing environment space, the illumination device 27 The illumination control data estimated values of the illumination devices ID1 to ID8 corresponding to a plurality of illumination units overlapping with the illumination range by the weighted average according to the area ratio of the region overlapping with the illumination range by the illumination device 27 are obtained. Illumination control data for 27 is calculated.

  For example, when the illumination range by the illumination device 27 installed in the actual viewing environment space overlaps with the illumination units (illumination ranges) corresponding to the illumination devices ID2, ID4, ID6, and ID8 in the virtual viewing environment space, Illumination control data estimated values obtained for the illumination devices ID2, ID4, ID6, and ID8 in the viewing environment space are represented by the illumination units corresponding to the illumination devices ID2, ID4, ID6, and ID8 and the illumination range by the illumination device 27. Is weighted average according to the area ratio of the overlapping region to obtain illumination control data for the illumination device 27.

  On the other hand, when the sizes of the viewing environment spaces do not match, the actual viewing environment space is converted so as to match the size of the virtual viewing environment space. Specifically, by changing the magnification in the vertical, horizontal, and height directions of the actual viewing environment space to match the size of the virtual viewing environment space, the size of both viewing environment spaces Can be matched.

  By matching the sizes of both viewing environment spaces, as in the case where the sizes of both viewing environment spaces match, the illumination control data estimation corresponding to each of the lighting devices ID1 to ID8 in the virtual viewing environment space is performed. From the value, the illumination control data for the illumination device 27 installed in the actual viewing environment space can be calculated. By performing the above calculation for other lighting devices 27 existing in the actual viewing environment space, it is possible to generate illumination control data for all the lighting devices 27 installed in the actual viewing environment space.

  In the above-described example, the illumination unit (illumination) corresponding to the illumination range of the illumination device 27 installed in the actual viewing environment space and the illumination devices ID1 to 8 of the illumination devices ID1 to 8 in the virtual viewing environment space. Although the illumination control data for the illumination device 27 is obtained according to the overlapping area with the range), it is obvious that other methods may be used as the calculation method of the illumination control data to be output to the illumination device 27.

  For example, the virtual viewing environment space is divided into eight space blocks surrounded by lighting units corresponding to the respective lighting device IDs 1 to 8 and corresponds to the space block where the lighting device 27 is located in the actual viewing environment space. Lighting control data of the lighting device 27 may be obtained using the lighting control data estimated values of the lighting devices ID1 to ID8.

  Further, another example of a method for calculating illumination control data to be output to the illumination device 27 installed in the actual viewing environment space will be described. The illumination control data for controlling the illumination device 27 in the actual viewing environment space is the illumination control of the illumination devices ID1 to 8 installed at the four corners of the wall surface of the virtual viewing environment space closest to the illumination device 27. Illumination control data for the lighting device 27 is calculated by performing a weighted average of the data estimation values according to the distance between the lighting device 27 and each of the lighting device IDs 1 to 8.

  For example, when the lighting devices ID1, ID3, ID5, and ID7 are installed at the four corners of the wall surface of the virtual viewing environment space closest to the lighting device 27 installed in the actual viewing environment space, The reciprocal ratio of the distance between the lighting device 27 and the lighting devices ID1, ID3, ID5, and ID7 is obtained, and the contribution ratio of each of the lighting devices ID1, ID3, ID5, and ID7 to the lighting device 27 is determined according to the reciprocal ratio. And the illumination control data with respect to the illuminating device 27 is obtained by performing the weighted average calculation according to the said contribution rate with respect to the illuminating control data estimated value calculated | required with respect to each of illuminating device ID1, ID3, ID5, and ID7. calculate.

  Hereinafter, a specific calculation method will be described. If the distances from the illumination devices 27 of the illumination devices ID1, ID3, ID5, and ID7 in the virtual viewing environment space are 1, 3, 2, and 4, respectively, the reciprocal ratio of the distances is 24: 8: 12: 6, respectively. . Here, in order to simplify the description, the distance between the lighting device 27 and the lighting devices ID1, ID3, ID5, and ID7 is described using the numerical values as described above as an example. Needless to say.

  Furthermore, the illumination control data estimated values for the illumination device ID1 are R = 250, G = 230, B = 150, the illumination control data estimated values for the illumination device ID3 are R = 170, G = 200, B = 100, the illumination device. The illumination control data estimated value for ID5 is R = 90, G = 110, B = 130, and the illumination control data estimated value for illumination device ID7 is R = 150, G = 50, B = 40.

  Next, illumination control data for the illumination device 27 is obtained from the reciprocal ratio of the distances from the illumination devices 27 of the illumination devices ID1, ID3, ID5, and ID7 and the estimated value of the illumination control data. First, the optimum value of the red light source (R) is calculated. The R values of the respective lighting devices ID1, ID3, ID5, and ID7 are ID1: 250, ID3: 170, ID5: 90, and ID7: 150.

  When this value is multiplied by the reciprocal ratio of the distances from the illumination devices 27 of the respective illumination devices ID1, ID3, ID5, and ID7, ID1: 250 × 24 = 6000, ID3: 170 × 8 = 1360, ID5: 90 × 12 = 1080 , ID7: 150 × 6 = 900, and the obtained value is added (6000 + 1360 + 1080 + 900 = 9340) and divided by the sum of the reciprocal ratio of distances (24 + 8 + 12 + 6 = 50) (9340/50 = 186.8). The illumination control data R for the red light source (R) can be calculated.

  When the illumination control data G for the green light source (G) and the illumination control data B for the blue light source (B) are obtained in the same manner as described above, the illumination control data (R: 187, G for appropriately controlling the illumination device 27). : 175, B: 124) can be calculated. By performing this calculation for other lighting devices 27 existing in the actual viewing environment space, it is possible to generate illumination control data for all the lighting devices 27 installed in the actual viewing environment space.

  In the above example, weighted interpolation is performed using the reciprocal ratio of the distance between the lighting device 27 installed in the actual viewing environment space and the lighting devices ID1 to 8 in the virtual viewing environment space. Obviously, other weighting methods may be used.

  Further, here, the illumination control data for the illumination device 27 installed in the actual viewing environment space is transferred to the four illumination devices ID1 to 8 on the wall surface in the virtual viewing environment space closest to the illumination device 27. Although it calculated | required from the corresponding illumination control data estimated value, for example, you may obtain | require from the illumination control data estimated value with respect to eight illumination device ID1-8 in virtual viewing-and-listening environment space, or two or more illumination device ID1 of the vicinity. Lighting control data for each lighting device 27 installed in the actual viewing environment space may be obtained by performing a predetermined interpolation calculation from the lighting control data estimated values for .about.8.

  In this embodiment, in order to easily obtain a desired lighting effect, the lighting control data (R) for the lighting device 27 installed in the actual viewing environment space is used by using the simplified virtual space model as described above. , G, B), the present invention is not limited to the one using the above virtual space model, and the mapping process from the virtual space to the real space is not limited to the above. Needless to say.

  As described above, the illumination control data estimation unit 352 uses the video data, the feature amount of the audio data, the scene setting information and / or the shooting illumination information and / or the camera work information to perform an illumination estimation process suitable for displaying each segment. Do.

  Here, in the present embodiment, a mode for performing illumination estimation processing using feature amounts of video data and audio data (video reference mode), and feature amounts of video data and audio data are not used and are included in additional data. There are modes (an additional information reference mode) for performing illumination estimation processing only from scene setting information and / or photographing illumination information, and these modes can be arbitrarily switched and set by the viewer on the video receiving device side. ing.

  When the video reference mode is set, the illumination control data estimation unit 352 obtains an illumination control data estimated value using the feature amounts of the video data and audio data. As a method for estimating the state of ambient light at the time of shooting using the feature amounts of video data and audio data, various techniques including known ones can be used. Here, in addition to the feature amount of the video data, the feature amount of the audio data is also used for estimating the field (atmosphere) of each segment, but this further improves the estimation accuracy of the field (atmosphere). For this reason, the field (atmosphere) of the shooting scene may be estimated from only the feature amount of the video data or using the analysis result of the feature amount of the video data and the additional data.

  Here, as the feature amount of the video data, for example, color signals (Cb, Cr) and a luminance signal (Y) are extracted from a display image of a color television, and the three-color light used for the light source is obtained from the color signal and the luminance signal. Calculate the appropriate dimming illuminance ratio of (red light, green light, blue light), set the illuminance of the three color light according to the illuminance ratio, mix the three color light and use it as lighting control data, color TV The video displayed on the screen is divided into a plurality of parts, the average hue of the video segment corresponding to the portion controlled by the lighting device is detected, and the lighting control corresponding to the segment is performed based on the detected hue A data estimated value may be calculated. Further, as the feature amount of the audio data, a volume, an audio frequency, or the like can be used.

  In addition, when performing the illumination estimation process using the feature amounts of the video data and audio data, the screen area for detecting the feature amount of the video data may be limited using the camera work information. Furthermore, the illumination control data estimated value calculated from the feature values of the video data and audio data may be corrected using the illumination control data estimated value calculated from the scene setting information and / or the shooting illumination information.

  On the other hand, when the additional information reference mode is set, the illumination control data estimation unit 352 uses only the scene setting information and / or the imaging illumination information without using the feature amounts of the video data and the audio data. I do. By using the scene setting information and shooting illumination information, the color temperature and illumination intensity of the illumination light used at the time of video shooting can be estimated more accurately.

  The general operation of the illumination estimation process when the video reference mode is set will be described with reference to FIG. In FIG. 16, first, the illumination control data estimation unit 352 determines the presence or absence of camera work information (S1). If there is camera work information, an illumination estimation process (illumination estimation process 1)) based on the camera work information and the video feature amount is performed (S2). After performing the illumination estimation process 1), the number of additional data is incremented and stored in the illumination control data estimation unit 352 (S3), and the presence / absence of the next additional data is confirmed (S1). Note that the number of additional data is incremented by 1 every time the number is counted by setting a default value of 0. If there is no camera work information, the presence / absence of photographing illumination information is determined (S1).

  If there is photographic illumination information, an illumination estimation process (illumination estimation process 2)) based on the photographic illumination information and the video feature amount is performed (S2). After performing the illumination estimation process 2), the number of additional data is incremented and stored in the illumination control data estimation unit 352 (S3), and the presence / absence of the next additional data is confirmed (S1). If there is no shooting illumination information, the presence / absence of scene setting information is determined (S1).

  If there is scene setting information, an illumination estimation process (illumination estimation process 3)) based on the scene setting information and the video feature amount is performed (S2). After performing the illumination estimation process 3), the number of additional data is incremented and stored in the illumination control data estimation unit 352 (S3), and the presence / absence of the next additional data is confirmed (S1). If there is no additional data corresponding to the segment, illumination correction processing is performed (S4).

  In the illumination correction process, the estimated illumination values obtained by the illumination estimation processes 1) to 3) are added and then divided by the number of additional data stored in step S3. Here, the illumination correction processing is performed by simply averaging the illumination estimation values obtained by the illumination estimation processing 1) to 3), but it goes without saying that the correction method is not limited to the above, such as performing a weighted average calculation. .

  In addition, the estimated illumination value by the illumination estimation process 1) is RGB data, and the estimated illumination value by the illumination estimation process 2) and 3) is the brightness and color temperature data of the illumination. Although it is necessary to convert the brightness and color of illumination to data that can be expressed and to perform correction calculation on the same data, data that can express brightness and color is not limited, and L * a * b * and Luv It may be in a uniform color space.

  The illumination estimated value subjected to the illumination correction processing as described above is output to the illumination control unit 354 as the illumination control data estimated value of the segment. The lighting control unit 354 obtains and outputs lighting control data for the lighting device 27 installed in the actual viewing environment space from the lighting control data estimated value. Accordingly, illumination light can be irradiated in consideration of the feature amount of the video data displayed on the video display device 25.

  Next, a schematic operation of the illumination estimation process when the additional information reference mode is set will be described with reference to FIG. In FIG. 17, the illumination control data estimation unit 352 first determines the presence or absence of photographing illumination information (S1). If there is photographic illumination information, an illumination estimation process (illumination estimation process 4)) using the photographic illumination information is performed (S2).

  After performing the illumination estimation process 4), the presence / absence of scene setting information is confirmed (S3). If there is scene setting information, illumination estimation processing (illumination estimation processing 5)) using the scene setting information is performed (S4). If it is confirmed in step S3 that there is no scene setting information, the color temperature data is converted into RGB data as the estimated illumination value obtained by the illumination estimation process 4) (S7).

  Then, after performing the illumination estimation process 5), an illumination estimation correction process is performed (S5). Here, a simple average of the estimated illumination value obtained by the illumination estimation process 4) and the estimated illumination value obtained by the illumination estimation process 5) is obtained, but the correction process considering visual characteristics in a uniform space Needless to say, it may be done. Since the illumination estimated value obtained by this illumination correction process is color temperature data, it is converted into RGB data (S7).

  If it is confirmed in step S1 that there is no shooting illumination information, the presence / absence of scene setting information is determined (S3). If there is scene setting information, an illumination estimation process using the scene setting information ( After performing the illumination estimation process 5)) (S4), conversion from color temperature data to RGB data is performed as the illumination estimation value obtained by the illumination estimation process 5) (S7).

  If it is confirmed in step S3 that there is no scene setting information, the illumination estimation process using only the additional data cannot be performed, so an error is displayed and the user is prompted to switch the setting to the video reference mode ( S8).

  The RGB data obtained in step S7 is output to the illumination control unit 354 as the illumination control data estimated value of the segment. The lighting control unit 354 obtains and outputs lighting control data for the lighting device 27 installed in the actual viewing environment space from the lighting control data estimated value. As a result, it is possible to irradiate illumination light that is substantially the same as the illumination light at the time of photographing video data displayed on the video display device 25.

  Next, illumination estimation processing 1) using video data and camera work information will be described with reference to the flowchart of FIG. First, the additional data analysis unit 351 reads a newly input segment and detects whether camera work information is added to the segment (S1). When camerawork information is detected, an illumination estimation area is extracted using this camerawork information (S2). Then, illumination estimation processing is performed from the extracted video data of the illumination estimation region (S3). If camera work information is not detected in step S1, illumination estimation processing based on video data is performed without extracting the illumination estimation region (S3).

  Here, a method for extracting the illumination estimation region will be described below. Here, based on the camera work information, the screen area used when detecting the feature amount of the video data is determined. In the present embodiment, the illumination estimation process is basically performed based on the video feature amount of the first frame of each segment.

  However, for example, in a frame in a segment whose subject size is one of bust shot, up shot, close-up shot, and big close-up shot, the subject (foreground part) occupies a large area area on the screen. The illumination estimation process using the video feature amount of the frame included in the segment is not performed, and the estimated illumination value obtained for the immediately preceding segment is continuously maintained and output.

  Also, the frame in a segment whose lens type is either a telephoto lens or a macro lens also uses the video feature amount of the frame included in the segment because the subject (foreground part) occupies a large area on the screen. The estimated illumination value obtained for the immediately preceding segment is continuously maintained and output without performing the estimated illumination estimation process.

  In addition, the frame in a segment where the camera movement can be pan, tilt (tilt), roll, zoom, dolly, or follow can have an extremely changing background as the camera moves. Therefore, the illumination estimation process using the video feature amount of the frame included in the segment is not performed, and the illumination estimation value obtained for the immediately preceding segment is continuously maintained and output.

  In addition, even in the first frame of each segment used for detecting the video feature amount, the screen area to be detected for the video feature amount differs depending on the camera work at the time of video shooting. For example, as shown in FIG. 19, a screen region suitable for estimating the color and brightness of ambient light at the time of shooting based on the camera position and angle at the time of shooting the video of the segment (in the shaded portion in FIG. 19). Determined). In other words, the screen area suitable for estimating the color and brightness of ambient light at the time of shooting is usually the area at the top of the screen, but the horizontal line that defines the size of the area at the top of the screen shows the camera position and angle. It is decided by.

  For example, as shown in FIGS. 20 and 21, it is suitable for estimating the color and brightness of ambient light at the time of shooting based on only the information on the camera position and the information on the angle at the time of video shooting of the segment. Needless to say, the screen area (indicated by the hatched portion in FIGS. 20 and 21) is determined.

  In addition, as shown in FIG. 22, a screen region suitable for estimating the color and brightness of ambient light at the time of shooting based on the subject size and number at the time of video shooting of the segment (the hatched portion in FIG. 22). Is determined). In other words, the screen area suitable for estimating the color and brightness of the ambient light at the time of shooting is desirably an area excluding the foreground subject, and this subject is highly likely to be located. The area to be excluded from the area is determined by the subject size and the number.

  For example, as shown in FIG. 23 and FIG. 24, a screen suitable for estimating the color and brightness of ambient light at the time of photographing based on information on only the subject size at the time of photographing the frame and information on only the number of people. Needless to say, the region (indicated by the hatched portion in FIGS. 23 and 24) is determined. In addition, when the number of subjects is only the background, the entire screen area is set as the detection target of the video feature amount.

  Therefore, in the present embodiment, the target area for detecting the video feature amount is limited based on the camera work information of the position, angle, subject size, and subject image. Specifically, the video feature is obtained by using only the video data of the screen area obtained by the logical product of the hatched area in FIG. 19 determined by the camera position and the camera angle and the hatched area in FIG. 22 determined by the subject size and the subject image. The amount is going to be detected.

Here, FIG. 25 shows a basic relationship between each of the lighting devices ID1 to ID8 in the virtual space described above with reference to FIG. 15 and the screen area for detecting the video feature amount. When the illumination estimation is performed only from the video feature amount, the illumination estimation is performed for each of the lighting devices ID1 to 8 in the virtual space from the video feature amount of the screen area illustrated in FIG. For example, for the illumination device ID1 located at the vertex V ₁ of the virtual space, as shown in FIG. 25A, the illumination control estimation data is obtained using the video feature amount in the upper left area of the screen.

  Further, when the screen area for detecting the video feature amount is limited using the camera work information, the logical area between the screen area extracted based on the camera work information and the screen area shown in FIG. 25 is obtained. The illumination control data estimated value for each of the lighting devices ID1 to 8 in the virtual space is calculated.

That is, by performing an AND operation on the screen area indicated by hatching in FIG. 25 and the target area shown in FIGS. 19 to 24, the brightness and color of the irradiation light by the respective illumination devices ID1 to 8 in the virtual space. Can be obtained. Note that the screen area indicated by hatching in FIG. 25, the AND operation to the target region shown in FIG 19 to 24, if the target area is not required for the lighting device ID5 located vertex V _5, as shown in FIG. detecting the image feature from the target area with respect to the illumination device ID1 located vertex V ₁ as shown in 25 (a).

Also, if the target area with respect to the illumination apparatus ID6 located vertex V ₆ is not required to detect the image feature from the target area with respect to the illumination apparatus ID2 located vertex V ₂ shown in FIG. 25 (b). If the target area for the lighting device ID7 located vertex V ₇ is not required to detect the image feature from the target area with respect to the illumination apparatus ID3 located vertex V ₃ shown in FIG. 25 (c). If the target area for the lighting device ID8 located vertex V ₈ is not required to detect the image feature from the target area with respect to the illumination apparatus ID4 located vertex V _4.

  In this way, the camera work information is used to restrict and extract frames and screen areas suitable for estimating the scene (atmosphere) of the video scene, and to detect the video feature quantities in the extracted frames and screen areas. The estimation accuracy of (atmosphere) can be improved. As a result, it is possible to prevent an illumination control that impairs the sense of presence or the atmosphere due to an estimation error of the place (atmosphere), and to always realize an optimal viewing environment.

  Further, the field (atmosphere) estimation process for each of the video scene examples shown in FIGS. 37 to 41 will be described. Here, it is assumed that the video scene shown in FIG. 37 is divided into a segment composed of frames A to E-1 and a segment composed of frame E.

  For the segment composed of the frames A to E-1, video data of a screen area in which the area indicated by the oblique lines in FIG. 19 (g) and the area indicated by the oblique lines in FIG. The video feature amount is detected. For the segment consisting of the frame E, the video data of the screen area where the area indicated by the oblique lines in FIG. 19G and the area indicated by the oblique lines in FIG. Detect video feature quantities. As a result, it is possible to generate an estimated value of illumination control data using only the video feature amount of the background portion, and to transmit appropriate viewing environment illumination light reflecting the illumination state at the time of photographing the scene over the display reproduction period of this scene. Irradiation is possible.

  In the video scene shown in FIG. 38, the camera work is divided into segments in different shot units. For the segments consisting of frames A to C-1 and the segments consisting of frames E to F, the areas indicated by the diagonal lines in FIG. 19 (e) and the areas indicated by the diagonal lines in FIG. 22 (b) in frames A and E. A video feature amount is detected using video data in a screen area where and overlap. Since the subject size is a bust shot for the segment composed of the frames C to E-1, the video data feature amount is not detected, and is obtained for the segment composed of the immediately preceding frames A to C-1. Continue to output the lighting control data estimate. Accordingly, it is possible to generate an estimated value of illumination control data using only the video feature amount of the background portion, and it is possible to irradiate appropriate viewing environment illumination light reflecting the illumination state at the time of photographing the scene.

  In the video scene shown in FIG. 39, since zoom shooting is performed, one segment is formed by frames A to E. For this segment, video data feature amount detection is performed. Instead, the illumination control data estimated value obtained for the immediately preceding segment is continuously output. Usually, there is a segment whose camera motion is fixed immediately before zoom shooting is started, so by using the illumination control data estimated value obtained for this segment, the lighting situation at the time of this scene shooting It is possible to irradiate the appropriate viewing environment illumination light reflecting the above over the display reproduction period of this scene.

  In the video scene shown in FIG. 40, since zoom photography is also performed, one segment is formed by frames A to E. For this segment, the detection of the video data feature amount is performed. Without performing this, the illumination control data estimated value obtained for the immediately preceding segment is continuously output. Also in this case, since there is usually a segment whose camera motion is fixed immediately before zoom shooting is started, this scene shooting can be performed by using the illumination control data estimated value obtained for the segment. It is possible to irradiate with appropriate viewing environment illumination light reflecting the lighting conditions of the time over the display reproduction period of this scene.

  In the video scene shown in FIG. 41, a segment composed of frames A to D-1 whose lens type is a standard lens and a frame D to E which is a bust shot when the lens type is switched to a telephoto lens. Are divided into segments and segments. For the segment composed of the frames A to D-1, the video data of the screen area where the area shown by the oblique line in FIG. 19 (e) and the area shown by the oblique line in FIG. Detect feature values. For the segment composed of the frames D to E, the video data feature quantity is not detected, and the illumination control data estimated value obtained for the segment composed of the immediately preceding frames A to D-1 is continuously output. As a result, it is possible to generate illumination control data using only the video feature amount of the background portion, and irradiate the appropriate viewing environment illumination light reflecting the illumination state at the time of shooting the scene over the display reproduction period of this scene. It becomes possible.

  As described above, in the present embodiment, the illumination control data is used to appropriately reproduce the illumination state (atmosphere) at the time of shooting each scene by using various information contents input as camera work information together with the video data. Thus, it is possible to control the illumination light of the illumination device 27. Therefore, it is possible to realize a natural and uncomfortable viewing environment illumination without being affected by the video content of the foreground part (subject) and the like, and it is possible to increase a sense of reality when viewing the video. In addition, since the lighting control data is switched and controlled in arbitrary segment units, it is possible to prevent the viewing environment lighting from changing drastically in the same segment and impairing the presence.

  The screen area patterns shown in FIGS. 19 and 22 can be set as appropriate. In addition, the histogram (frequency distribution) of the video data in the field (atmosphere) estimation screen area obtained from this screen area pattern is detected, and the video feature quantity is detected only from the video data with a high distribution ratio. ) By performing the estimation, the accuracy of the field (atmosphere) estimation can be improved.

  In this embodiment, since camera work information related to the camera work status at the time of video shooting of each segment is transmitted and received, a desired segment can be searched and edited using this camera work information. Various functions can be realized in addition to control of environmental lighting.

  For example, camera position is eye height, angle is horizontal angle, subject size (screen size) is long shot, camera movement is fixed, lens type is to select / extract the first frame of a segment shot using a standard lens This makes it possible to create an index that collects representative video scenes (key frames, key shots).

  Next, the illumination estimation process 4) using the photographic illumination information will be described with reference to the flowchart of FIG. First, the additional data analysis unit 351 reads a newly input segment and detects whether or not photographing illumination information is added to the segment (S1). When photographing illumination information is detected, illumination estimation processing is performed using the photographing illumination information. If the photographic illumination information is not detected, the flow ends without performing the illumination estimation process.

  The illumination control data estimation unit 352 uses each of the illumination devices ID1 to ID1 in the virtual space to reproduce the illumination environment at the time of video shooting of each scene by the viewing environment illumination by the illumination device 27 based on the imaging illumination information described above with reference to FIG. The intensity and color temperature of illumination light (one point on the black body locus) are determined every 8 and output to the illumination control unit 354 as illumination control data estimated values. Here, based on various types of information included in the photographic illumination information, for example, as shown in FIG. 27, control data estimated values indicating the illumination intensity in 7 levels and the illumination color temperature in 16 levels are set for each illumination device in the virtual space. Generated for each ID. The illumination control data estimation unit 352 adds control information for each lighting type to generate control data for each ID of each lighting device in order to reproduce the lighting environment at the time of video shooting.

  Here, a black body refers to an ideal object that completely absorbs energy, and when the temperature rises, the color of the emitted light changes from red to yellow to white. The absolute temperature at this time is called color temperature. When this color temperature and color locus (black body locus) are represented on the xy chromaticity diagram, it is as shown in FIG. When the color of the light source is not on the black body locus, the temperature of the closest black body that does not completely match is called “correlated color temperature”. In general, the correlated color temperature is expressed together with the deviation (Δuv) from the black body locus.

  For example, when the illumination property information is a spotlight (point light source), illumination light having a predetermined intensity and color temperature is obtained by N illumination devices centered on an illumination device having an ID located in the incident direction of the illumination light. Shall be irradiated. In addition, when the illumination property information is a flood light (surface light source), a predetermined intensity and a wide range of M (M >> N) illumination devices including ID illumination devices located in the incident direction of the illumination light. It shall be irradiated with illumination light of color temperature.

  Here, an example of the relationship between the incident direction of the photographing illumination light and the illumination devices ID1 to ID8 in the virtual space is as shown in FIG. 28, for example. That is, when the direction of the illumination light in the horizontal direction is forward light and the direction of the illumination light in the vertical direction is other than the side, the two illumination devices ID1 and ID2 are used to reproduce the illumination light. In addition, when the direction of the illumination light in the horizontal direction is side light (left) and the direction of the illumination light in the vertical direction is other than the tilt, in order to reproduce this illumination light, the illumination devices ID1 and ID3 are connected in the horizontal direction. In the case where the direction of the illumination light is side light (right) and the direction of the illumination light in the vertical direction is other than the tilt, the illumination devices ID2 and ID4 are used to reproduce the illumination light.

  Further, in the case where the direction of the illumination light in the horizontal direction is the reverse light and the direction of the illumination light in the vertical direction is other than the tilt, in order to reproduce this illumination light, the illumination devices ID1 and ID2 are changed to the direction of the illumination light in the horizontal direction. Is the rim light (left), and the direction of the illumination light in the vertical direction is other than the tilt, in order to reproduce this illumination light, the illumination device ID1 is used, the direction of the illumination light in the horizontal direction is the rim light (right), When the direction of the illumination light in the vertical direction is other than tilt, the illumination device ID2 is used to reproduce this illumination light.

  In addition, when the direction of the illumination light in the horizontal direction is Rembrandt light (left) and the direction of the illumination light in the vertical direction is other than the tilt, in order to reproduce this illumination light, the illumination device ID1 is set to the illumination in the horizontal direction. When the direction of the light is Rembrandt light (right) and the direction of the illumination light in the vertical direction is other than vertical, the illumination device ID2 is used to reproduce this illumination light.

  Also, in the case where the direction of the illumination light in the horizontal direction is forward light, and the direction of the illumination light in the vertical direction is vertical, in order to reproduce this illumination light, the illumination devices ID5 and ID6 are connected to the illumination light in the horizontal direction. When the direction is side light (left) and the direction of illumination light in the vertical direction is vertical, in order to reproduce this illumination light, the illumination devices ID5 and ID7 are used, and the direction of the illumination light in the horizontal direction is side light (right). When the direction of the illumination light in the vertical direction is tilted, the illumination devices ID6 and ID8 are used to reproduce the illumination light.

  Furthermore, when the direction of the illumination light in the horizontal direction is the reverse light and the direction of the illumination light in the vertical direction is tilted, in order to reproduce this illumination light, the illumination devices ID5 and ID6 are used, and the direction of the illumination light in the horizontal direction is In the case of a rim light (left) and a vertical illumination light direction, the illumination device ID5 is used to reproduce this illumination light, and the horizontal illumination light direction is a rim light (right) and the vertical direction. When the direction of the illumination light is tilted, the illumination device ID 6 is used to reproduce the illumination light.

  In addition, when the direction of the illumination light in the horizontal direction is a Rembrandt light (left) and the direction of the illumination light in the vertical direction is vertical, in order to reproduce this illumination light, the illumination device ID5 is used. Is a Rembrandt light (right) and the direction of illumination light in the vertical direction is vertical, the illumination device ID 6 is used to reproduce this illumination light.

  In addition, when the direction of the vertical illumination light is the top light regardless of the direction of the horizontal illumination light, four illumination devices ID1 to ID4 are used to reproduce the illumination light. In this way, the illumination control data estimated values corresponding to the plurality of lighting devices arranged in the virtual space are determined based on the photographing illumination information, and installed in the actual viewing environment space based on the illumination control data estimated values. By independently controlling the plurality of illuminating devices 27, it is possible to reproduce the lighting conditions at the shooting site in detail and increase the sense of reality when viewing the video.

  Further, a specific example of lighting control according to the photographing lighting information will be described below. FIG. 29 shows photographing illumination information corresponding to the video scene described above with reference to FIG. 37. In this video scene, the illumination light source has a spotlight (point light source) property, the horizontal illumination light incident direction is forward light, and the vertical direction. The incident direction of the illumination light is a top light, the illumination intensity is strong, and the illumination color temperature is about 6000K.

  The illumination control data estimation unit 352 has a strong illumination intensity and an illumination color temperature of about 6000K (on the xy chromaticity diagram of FIG. 30) by the illumination devices ID1 to ID4 arranged in the virtual space based on the photographing illumination information. , (B) Illumination control data estimation values that generate illumination light (indicated by points) are generated.

  The illumination control unit 354 receives the illumination control data estimated value from the illumination control data estimation unit 352, calculates and outputs illumination control data (RGB data) for the illumination device 27 installed in the actual viewing environment space. As a result, it is possible to irradiate the surroundings of the video display device 25 with appropriate illumination light, and to reproduce the illumination environment at the time of shooting the above-described display video scene in the viewing space, thereby increasing the sense of reality when viewing the video. be able to.

  Further, FIG. 31 shows shooting illumination information corresponding to the video scene described above with reference to FIG. 38. In this video scene, the illumination light source has a floodlight (surface light source) property, and the horizontal illumination light incident direction is front light. The vertical illumination light incident direction is top light, the illumination intensity is normal, the illumination color temperature is about 7000K, the illumination light source is flood light (surface light source), and the horizontal light incidence direction is Rembrandt Light (left), vertical light incident direction, illumination intensity is weak, illumination color temperature is about 7000K fill light, illumination light source is spotlight (point light source), horizontal light incidence The direction is side light (right), the incident direction of vertical light is forward light, the illumination intensity is slightly weak, and the image is taken under an illumination condition with a touch light having an illumination color temperature of about 5000K.

  The illumination control data estimation unit 352 has a normal illumination intensity and an illumination color temperature of about 7000 K (on the xy chromaticity diagram of FIG. The illumination light of (c) (shown by a point) is irradiated, and the illumination device ID2 has a slightly strong illumination intensity and an illumination color temperature of about 7000K (shown by the point (c) on the xy chromaticity diagram of FIG. 30). An illumination control data estimated value that irradiates illumination light is generated.

  Further, the illumination device ID4 emits illumination light with a weak illumination intensity and an illumination color temperature of about 7000K (indicated by point (c) on the xy chromaticity diagram of FIG. 30), and the illumination device ID5 provides illumination intensity. However, the illumination control data estimated value is generated so as to irradiate illumination light having a slightly weak illumination color temperature of about 5000K (indicated by point (a) on the xy chromaticity diagram of FIG. 30).

  In the above-described example, the illumination control data estimation value that illuminates the illumination light obtained by adding the illumination light from the key light and the touch light is generated by the illumination device ID2, but the plurality of illumination lights are added together. In this case, by adding the illumination intensity of each illumination light, obtaining the intensity of illumination light emitted from the illumination device, and weighted averaging the illumination color temperature of each illumination light according to the illumination intensity ratio, What is necessary is just to obtain | require the illumination color temperature of the illumination light irradiated from the said illuminating device (Here, illumination control data showing about 7000K nearest to the result of the weighted average calculation as the illumination color temperature of the illumination light by illumination device ID2. Generating an estimate).

  As described above, in the present embodiment, from various information contents input as shooting illumination information, an illumination control data estimated value is obtained to appropriately reproduce the lighting situation (atmosphere) at the time of video shooting of each segment. It is possible to output illumination control data for controlling the illumination light of the illumination device 27 installed in the actual viewing environment space according to the estimated value of the illumination control data. Therefore, it is possible to realize a natural and uncomfortable viewing environment illumination without being affected by the video content, and to increase the sense of reality when viewing the video.

  In the present embodiment, since shooting illumination information related to the lighting situation during video shooting of each segment is transmitted and received, viewing environment illumination control adapted to various viewing environment spaces can be performed. That is, by appropriately inputting information related to the viewing space / illumination device such as the number and arrangement of the illumination devices to the illumination control unit 354, appropriate control data corresponding to each video viewing space can be obtained from the received photographing illumination information. Can be generated.

  In addition, when there is only one lighting device or when independent control of a plurality of lighting devices is impossible, the viewing environment illumination may be controlled using, for example, only information relating to the keylight. In this way, an optimal video viewing space can be realized by appropriately selecting / processing the received shooting illumination information according to the viewing space / lighting device and controlling the viewing environment illumination. In this case, the photographing illumination information can be selected / required by preparing and transmitting / receiving the illumination status information regarding each of the keylight, filllight, touchlight, and baselight as a data structure having a hierarchical structure. It is possible to process.

  As described above, the illumination estimation process 4) at the time of setting the additional information reference mode has been described in detail, but the illumination estimation process 2) at the time of setting the video reference mode is based on the illumination control data estimated value obtained by the above-described illumination estimation process 4). This is realized by calculating an estimated value of the illumination control data taking into account the feature value detection result of the video data.

  For example, information indicating a predetermined color range including the intensity and color temperature of illumination light (one point on the black body locus) and each color temperature is obtained for each illumination device ID 1 to 8 in the virtual space based on the photographing illumination information. The illumination control data estimation result obtained based on the video data and the audio data is corrected so as to be within the color range obtained based on the photographing illumination information, and the illumination control data estimated value (RGB data) is obtained. You can do it.

  In addition, an illumination that outputs an average value or a weighted average value of the illumination control data estimation result obtained based on the video data and the audio data and the illumination control data estimation result obtained based on the photographing illumination information to the illumination control unit 354 A control data estimated value may be used. Needless to say, the above-described illumination control data estimated value is generated for each of the illumination devices ID 1 to 8 in the virtual space.

  As described above, by using the shooting illumination information and the video data and / or audio data to control the viewing environment illumination, the feature amount of the video data and / or audio data in addition to the shooting illumination information can be obtained. It is possible to perform illumination control in consideration of the estimated result of the field (atmosphere) based on the result. In addition, while realizing dynamic viewing environment illumination control in consideration of the feature amount of the video data and / or audio data, the presence of the field (atmosphere) estimation error based on the feature amount of the video data and / or audio data can be realistic. It is possible to prevent lighting control that impairs the feeling and atmosphere, and to realize an appropriate viewing environment.

  Next, illumination estimation processing 5) using scene setting information will be described with reference to the flowchart of FIG. First, the additional data analysis unit 351 reads a newly input segment and detects whether scene setting information is added to the segment (S1). If scene setting information is detected, illumination estimation processing is performed. If the scene setting information is not detected, the flow is terminated without performing the illumination estimation process.

  From the scene setting information described above with reference to FIG. 12, the lighting control data estimation unit 352 reproduces the setting scene (atmosphere) on the story of each scene by viewing environment illumination by the lighting device 27, and each lighting device ID1 in the virtual space. The intensity and color temperature (one point on the black body locus) of the illumination light are determined every ˜8, and this is output to the illumination control unit 354 as the illumination control data estimated value. Here, based on various kinds of information included in the scene setting information, for example, as shown in FIG. 33, the illumination control data estimated values indicating the illumination intensity in 7 levels and the illumination color temperature in 16 levels are shown in the virtual space. For each ID.

  Hereinafter, a specific example of lighting control according to the scene setting information will be described with reference to FIG. For example, if the season information in the scene setting information is spring, the time information is daytime, the location information is outdoors, and the weather information is clear, the illumination intensity is slightly higher and the illumination color temperature is 6000K (Figure) to reproduce the natural light at this time. 34, the illumination control data estimated value indicating that it is indicated by (d) point) is generated. Here, when the season information is summer, an illumination control data estimated value indicating that the illumination intensity is strong while the illumination color temperature remains at 6000K is generated.

  In addition, when the regional information is tropical, control data indicating that the illumination intensity is very strong is generated. Furthermore, when the seasonal information is autumn, the illumination control data estimated value indicating that the illumination intensity is slightly strong, and when the seasonal information is winter, the illumination intensity is normal and the illumination color temperature is 7000 K (at point (e) in FIG. 34). Control data is generated to indicate that there is. Here, when the area information is a cold zone, an illumination control data estimated value indicating that the illumination intensity is slightly weak is generated.

  Also, if the time information in the scene setting information is morning, the location information is indoors, and the weather information is clear, the illumination intensity is normal and the illumination color temperature is 4000K (in order to reproduce the situation where natural light is inserted indoors at this time. An illumination control data estimated value indicating that (indicated by point (b) in FIG. 34) is generated. Here, when the time information is evening, the illumination control data estimated value indicating that the illumination intensity is normal and the illumination color temperature is 3000K (indicated by point (a) in FIG. 34) is generated. In addition, when the time information becomes night, the illumination intensity is slightly weak and the illumination color temperature is 5000K (indicated by point (c) in FIG. 34) in order to reproduce the illumination light of the fluorescent lamp (lunch white) that is an indoor illumination device. An illumination control data estimated value indicating that is generated.

  Furthermore, if the time information in the scene setting information is night, the location information is indoors, and the period information is a period drama, the candle flame is likely to be an illumination light source, and the illumination intensity is slightly weaker to reproduce this. The illumination control data value indicating that the illumination color temperature is 3000K (indicated by point (a) in FIG. 34) is generated. Here, when the location information is outdoors, there is a high possibility that moonlight is an illumination light source, and in order to reproduce the atmosphere of the moonlit night of a historical drama, the illumination intensity is very weak and the illumination color temperature is 7000K (in FIG. 34 ( e) Illumination control data estimated values are generated, which are indicated by points).

  Also, if the time information in the scene setting information is daytime, the location information is outdoors, and the weather information is fog, haze, haze, or snow, the illumination intensity is weak and the illumination color temperature is 6000K to reproduce the natural light at this time. An illumination control data estimated value indicating that (indicated by point (d) in FIG. 34) is generated. If the combination of various types of information described above with reference to FIG. 33 is not met, it is assumed that there is no scene setting information, and the illumination estimation process is terminated.

  In addition, the illumination control data estimated value calculated | required from scene setting information is good also as the illumination control data estimated value corresponding to all the illuminating device ID1-8 arrange | positioned in virtual space, or the illuminating device ID1 located upward. It is good also as the illumination control data estimated value corresponding to 4.

  As described above, in the present embodiment, the illumination control data estimated values corresponding to the illumination devices ID1 to 8 arranged in the virtual space are determined from the various information contents input as the scene setting information together with the video data, By controlling the lighting device 27 installed in the actual viewing environment space based on the estimated value of the lighting control data, the setting scene (atmosphere) on the story of each segment can be appropriately reproduced. Therefore, it is possible to realize a natural and uncomfortable viewing environment illumination without being affected by the video content, and to increase the sense of reality when viewing the video.

  In the present embodiment, scene setting information related to the setting scene on the story of each scene is transmitted and received. Therefore, the scene setting information is used to search for and edit a desired scene. Various data processing is possible in addition to the above control.

  As described above, the illumination estimation process 5) at the time of setting the additional information reference mode has been described in detail, but the illumination estimation process 3) at the time of setting the video reference mode is the illumination control data estimated value obtained by the above-described illumination estimation process 5). This is realized by calculating an estimated value of the illumination control data taking into account the feature value detection result of the video data.

  For example, based on the scene setting information, information indicating a predetermined color range including the intensity and color temperature of illumination light (one point on the black body locus) and each color temperature is obtained for each of the lighting devices ID1 to 8 in the virtual space. The illumination control data estimation result obtained based on the video data and the audio data is corrected so as to be within the color range obtained based on the scene setting information, and the illumination control data estimated value (RGB data) is obtained. You can do it.

  In addition, the illumination control unit 354 outputs an average value or a weighted average value between the illumination control data estimation result obtained based on the video data and the audio data and the illumination control data estimation result obtained based on the scene setting information to the illumination control unit 354. A control data estimated value may be used. Needless to say, the above-described illumination control data estimated value is generated for each of the illumination devices ID 1 to 8 in the virtual space.

  As described above, by using the scene setting information and the video data and / or audio data to control the viewing environment illumination, the feature amount of the video data and / or audio data can be added to the scene setting information. It is possible to perform illumination control in consideration of the estimated result of the field (atmosphere) based on the result. In addition, while realizing dynamic viewing environment illumination control in consideration of the feature amount of the video data and / or audio data, the presence of the field (atmosphere) estimation error based on the feature amount of the video data and / or audio data can be realistic. It is possible to prevent lighting control that impairs the feeling and atmosphere, and to realize an appropriate viewing environment.

  As described above, when the additional information reference mode is set, the color temperature and illumination intensity of the illumination light used at the time of video shooting can be estimated more accurately by using the scene setting information and the shooting illumination information. Thereby, the intensity of the viewing environment illumination light and the color temperature can be appropriately controlled to improve the sense of reality when displaying the video. In addition, when this additional information reference mode is set, it is possible to realize a viewing environment using natural illumination light regardless of the feature amount of the video data and audio data, so it is desirable to view the video in a calm atmosphere. Suitable for cases.

  On the other hand, when the video reference mode is set, the illumination control data estimated value is calculated using the feature values of the video data and audio data, so the color reproduction range taking into account the feature values of the video data and audio data is large. It is possible to control the dynamic viewing environment illumination light. When this additional information reference mode is set, this mode is suitable for stimulating video viewing with more emphasis on the power of video and audio.

  In the present embodiment, as described above, since the user can arbitrarily switch between the additional information reference mode and the video reference mode, the illumination estimation process is performed depending on the type of video content to be displayed, the preference of the viewer, and the like. The desired viewing environment illumination is realized by changing the.

  In the present embodiment, the illumination control data estimated values for the illumination devices ID1 to 8 in the virtual space shown in FIG. 15 are first obtained, and then the illumination control data for the illumination device 27 installed in the actual viewing environment space is obtained. Although it is calculated | required, you may make it obtain | require the illumination control data with respect to the illuminating device 27 installed in the actual viewing-and-listening environment space directly, without calculating | requiring the illumination control data estimated value with respect to illuminating device ID1-8 in virtual space. Needless to say.

  Further, in the present embodiment, the case where additional data such as segment delimiter information is multiplexed and added to the broadcast data has been described. However, when the additional data is not added to the broadcast data, the video data to be displayed is displayed. By transmitting and receiving the corresponding lighting control metadata from an external server device or the like, it is possible to realize an optimal viewing environment according to the lighting situation during video shooting. This will be described below.

<Second Embodiment>
Hereinafter, the second embodiment of the viewing environment control system of the present invention will be described in detail with reference to FIGS. 35 and 36. However, the same parts as those in the first embodiment are denoted by the same reference numerals, and description thereof will be omitted. Here, FIG. 35 is a block diagram showing the schematic configuration of the main part of the external server device in the viewing environment control system of the present embodiment, and FIG. 36 shows the schematic configuration of the main part of the video receiving device in the viewing environment control system of the present embodiment. It is a block diagram.

  As shown in FIG. 35, the external server device (data transmission device) in the present embodiment sends a transmission request for illumination control data such as camera work information, shooting environment information, and scene setting information from the video reception device (data reception device) side. The receiver 51 for receiving the data, the data storage 52 for storing the illumination control metadata, and the illumination control metadata for which the transmission request has been received, together with the segment delimiter information, the requesting video receiver (data receiver) And a transmitting unit 53 for transmitting to.

  Next, a video reception device (data reception device) that receives segment segment information and illumination control metadata sent from the external server device and controls viewing environment illumination will be described. As shown in FIG. 36, the video receiving apparatus in the present embodiment receives and demodulates broadcast data input from the transmission path, performs error correction, and outputs video from the output data of the receiving unit 61. A data separation unit 62 that separates and extracts each of the video data output to the display device 36 and the audio data output to the audio playback device 37, and a transmission request for lighting control metadata corresponding to the video data (content) to be displayed. A transmission unit 67 that transmits to an external server device (data transmission device) via a communication network, and a reception unit 68 that receives the illumination control metadata requested for transmission from the external server device via the communication network together with segment delimiter information. And.

  The segment delimiter information and the illumination control metadata received by the receiving unit 68 are temporarily stored. The start time TC (time code) included in the segment delimiter information and the TC of the video data extracted by the data demultiplexing unit 62 are stored. (Time code) are compared, and when they match, the CPU 66 outputs the corresponding lighting control data, and the lighting control metadata information from the CPU 66 and / or the feature amount of the video data and the audio data And an illumination control data generator 65 that generates illumination control data (RGB data) and outputs the illumination control data (RGB data) to the illumination device 38 that illuminates the viewing environment space.

  That is, the CPU 66 receives the start time code of each segment of the lighting control metadata storage table received from the external server device and stored therein, and the time code of the video data input to the lighting control data generation unit 65. When these are matched, the lighting control metadata corresponding to the segment is output to the lighting control data generation unit 65. The lighting control data generation unit 65 includes at least one of the segment delimiter information, camera work information, shooting environment information, and scene setting information included in the lighting control metadata, and / or video data and audio data. Is used to generate and output illumination control data (RGB data) suitable for displaying each frame. Since the operation of the illumination control data generation unit 65 is the same as that of the illumination control data generation unit 24 in the first embodiment, description thereof is omitted.

  As a result, even if the lighting control metadata is not added to the broadcast data, the lighting control metadata corresponding to the display video data (program content) is obtained from the external server device, and based on the lighting control metadata. Since the viewing environment illumination is controlled, appropriate illumination control according to the intention of the video producer can be performed as in the first embodiment.

  Note that the viewing environment control device, method, and viewing environment control system of the present invention can be realized by various embodiments without departing from the gist of the present invention described above. For example, the environmental lighting control device may be provided in the video display device, and it is needless to say that an external lighting device may be controlled based on various information included in the input video data. Yes.

  Furthermore, the segment delimiter information and the lighting control metadata (at least one of camera work information, shooting environment information, and scene setting information) described above are separated or obtained from broadcast data, or obtained from an external server device. For example, when displaying video information reproduced by an external device (such as a DVD player or a Blu-ray disc player), the segment delimiter information and the illumination control metadata recorded in the media medium are read out. May be used.

  In this case, the segment delimiter information and the illumination control metadata may be added to the video information, or may be recorded in a different area from the video information in the media medium. The segment delimiter information and lighting control metadata may be obtained from a media medium other than the medium on which the video information is recorded or an external server device. Needless to say, it is only necessary to be associated with the video data.

It is a block diagram which shows the principal part schematic structure of the video transmission apparatus in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows the example of an output bit stream of the video transmission apparatus in the viewing-and-listening environment control system of one Embodiment of this invention. It is a figure for demonstrating an example of the data structure of the additional data in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing for demonstrating the component of an image | video. It is explanatory drawing which shows an example of the scene break information in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows an example of the camera work information in the viewing-and-listening environment control system of one Embodiment of this invention. It is a figure for demonstrating a camera position and a camera angle. It is a figure for demonstrating a to-be-photographed object size. It is a figure for demonstrating the number of subjects. It is explanatory drawing which shows an example of the imaging | photography illumination information in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing for demonstrating the division of the horizontal direction of an imaging | photography illumination light, and a perpendicular direction. It is explanatory drawing which shows an example of the scene setting information in the viewing-and-listening environment control system of one Embodiment of this invention. It is a block diagram which shows the principal part schematic structure of the video receiver in the viewing-and-listening environment control system of one Embodiment of this invention. It is a block diagram which shows the structure of the illumination control data generation part in FIG. It is explanatory drawing which shows the virtual viewing environment space in the viewing environment control system of one Embodiment of this invention. It is a flowchart which shows the illumination estimation process at the time of the image | video reference mode in the viewing-and-listening environment control system of one Embodiment of this invention. It is a flowchart which shows the illumination estimation process at the time of the additional information reference mode in the viewing-and-listening environment control system of one Embodiment of this invention. It is a flowchart which shows the illumination estimation process using the camera work information in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows the example of a field (atmosphere) estimation object area | region decided by the camera position and camera angle in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows the example of a field (atmosphere) estimation object area | region decided by the camera position in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows the example of a field (atmosphere) estimation object area | region decided by the camera angle in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows the example of a field (atmosphere) estimation object area | region determined by the to-be-photographed object size and the to-be-photographed object in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows the example of a field (atmosphere) estimation object area | region decided by the to-be-photographed object size in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows the example of a field (atmosphere) estimation object area | region decided by the number of subjects in a viewing-and-listening environment control system. It is a figure which shows the screen feature-value detection area | region in the viewing-and-listening environment control system of one Embodiment of this invention. It is a flowchart which shows the illumination estimation process using the imaging | photography illumination information in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows an example of the control data in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows the example of a relationship between the illuminating device ID corresponding to the illumination direction information in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows the example of the imaging | photography illumination information corresponding to the video scene shown in FIG. It is explanatory drawing showing the color reproduction range of the illumination light in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows the imaging | photography illumination information corresponding to the video scene shown in FIG. It is a flowchart which shows the illumination estimation process using the scene setting information in the viewing-and-listening environment control system of one Embodiment of this invention. It is explanatory drawing which shows the example of a relationship between the scene setting information and control data in the viewing-and-listening environment control system of one Embodiment of this invention. It is another explanatory drawing showing the color reproduction range of the illumination light in the viewing-and-listening environment control system of one Embodiment of this invention. It is a block diagram which shows the principal part schematic structure of the external server apparatus in the viewing-and-listening environment control system which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the principal part schematic structure of the video receiver in the viewing-and-listening environment control system which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating an example of the problem of the illumination control by a prior art. It is a figure for demonstrating the other example of the problem of the illumination control by a prior art. It is a figure for demonstrating the other example of the problem of the illumination control by a prior art. It is a figure for demonstrating the other example of the problem of the illumination control by a prior art. It is a figure for demonstrating the other example of the problem of the illumination control by a prior art.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Data multiplexing part 2 ... Transmission part 11 ... Header 12 ... Extension header 13 ... Payload 21 ... Reception part 22 ... Data separation part 24 ... Illumination control data generation part 25 ... Video display apparatus 26 ... Audio | voice reproduction apparatus 27 ... Illumination apparatus 28 , 29 ... Delay generators 31 and 61 ... Receivers 32 and 62 ... Data separators 33 and 34 ... Delay generators 35 and 65 ... Illumination control data generator 36 ... Video display device 37 ... Audio reproduction device 38 ... Illuminator 51 ... Reception unit 52 ... Data storage unit 53 ... Transmission unit 61 ... Reception unit 62 ... Data separation unit 65 ... Lighting control data generation unit 66 ... CPU
67 ... Transmission unit 68 ... Reception unit 351 ... Additional data analysis unit 352 ... Illumination control data estimation unit 354 ... Illumination control unit

Claims (28)

In a data transmission apparatus for transmitting video data composed of one or more frames,
The video data is divided into segments composed of an arbitrary number of one or more frames, segment delimiter information indicating the start time and / or section of each segment, camera work information at the time of shooting the video data, and shooting illumination A data transmitting apparatus, wherein at least one of information and scene setting information is added to video data in units of segments. The data transmission device according to claim 1, wherein
The data transmission apparatus characterized in that the camera work information includes at least information representing a camera position at the time of photographing each segment. In the data transmission device according to claim 1 or 2,
The data transmission apparatus characterized in that the camera work information includes at least information representing a camera angle at the time of photographing each segment. In the data transmission device according to any one of claims 1 to 3,
The data transmission apparatus according to claim 1, wherein the camera work information includes at least information representing a size of a subject at the time of photographing each segment. In the data transmission device according to any one of claims 1 to 4,
The data transmission apparatus according to claim 1, wherein the camera work information includes at least information indicating the number of subjects at the time of photographing each segment. In the data transmission device according to any one of claims 1 to 5,
The data transmission apparatus according to claim 1, wherein the camera work information includes at least information representing a camera movement at the time of photographing each segment. In the data transmission device according to any one of claims 1 to 6,
The data transmission apparatus characterized in that the camera work information includes at least information indicating a type of a camera lens used for photographing each segment. In the data transmission device according to any one of claims 1 to 7,
The photographic illumination information includes at least information indicating a lighting type of illumination used for photographing each segment. In the data transmission device according to any one of claims 1 to 8,
The data transmission apparatus according to claim 1, wherein the photographing illumination information includes at least information indicating a property of illumination used for photographing each segment. In the data transmission device according to any one of claims 1 to 9,
The photographic illumination information includes at least information indicating the direction of illumination light used for photographing each segment. In the data transmission device according to any one of claims 1 to 10,
The photographic illumination information includes at least information indicating the intensity of illumination light used for photographing each segment. The data transmission device according to any one of claims 1 to 11,
The photographic illumination information includes at least information indicating the color temperature of the illumination light used for photographing each segment. The data transmission device according to any one of claims 1 to 12,
The data transmission device, wherein the scene setting information includes at least information indicating a season on a story of each segment. In the data transmission device according to any one of claims 1 to 13,
The data transmission device, wherein the scene setting information includes at least information representing a time on a story of each segment. In the data transmission device according to any one of claims 1 to 14,
The data transmission device, wherein the scene setting information includes at least information representing a place on a story of each scene. In the data transmission device according to any one of claims 1 to 15,
The data transmission device, wherein the scene setting information includes at least information representing an era on a story of each segment. In the data transmission device according to any one of claims 1 to 16,
The scene transmission information includes at least information representing the weather on the story of each segment. The data transmission device according to any one of claims 1 to 19,
The data transmission device, wherein the scene setting information includes at least information indicating a region on a story of each segment.   In response to an external request, segment delimiter information indicating the start time and / or section of a segment obtained by dividing video data into an arbitrary number of one or more frames, camera work information at the time of shooting the video data, and shooting illumination A data transmitting apparatus, wherein at least one of information and scene setting information is transmitted in units of segments. Video data to be displayed on a display device, segment delimiter information indicating the start time and / or section of a segment obtained by dividing the video data by an arbitrary number of one or more frames, and the video data in association with each segment Receiving means for receiving at least one of camera work information, shooting illumination information, and scene setting information at the time of shooting the video data;
Based on the segment delimiter information and at least one of the video data and / or camera work information, photographing illumination information, and scene setting information, illumination light of an illumination device installed around the display device is controlled. A viewing environment control apparatus comprising a control means.   21. The viewing environment control device according to claim 20, wherein the control unit switches and controls illumination light of the illumination device in units of the segments.   At least one of the video reference mode for controlling the illumination light of the lighting device using at least the feature amount of the video data, and the shooting illumination information and the scene setting information without using the feature amount of the video data. The viewing environment control device according to claim 20 or 21, further comprising: an additional information reference mode for controlling illumination light of the illumination device. The viewing environment control device according to claim 22,
When the video reference mode is set, the control unit restricts a screen area for detecting a feature amount of the video data according to the camera work information. In the viewing environment control device according to claim 22 or 23,
When the video reference mode is set, the control means determines the illumination obtained based on at least one of the estimated illumination value obtained based on the feature amount of the video data, the photographing illumination information, and the scene setting information. A viewing environment control device that controls illumination light of the illumination device using an estimated value.   25. A viewing environment control system comprising the viewing environment control device according to claim 20 and a lighting device whose viewing environment illumination light is controlled by the viewing environment control device. In a data transmission method for transmitting video data composed of one or more frames,
The video data is divided into segments composed of an arbitrary number of one or more frames, segment delimiter information indicating the start time and / or section of each segment, camera work information at the time of shooting the video data, and shooting illumination A data transmission method characterized by adding at least one of information and scene setting information to the video data in units of segments.   In response to an external request, segment delimiter information indicating the start time and / or section of a segment obtained by dividing video data into an arbitrary number of one or more frames, camera work information at the time of shooting the video data, and shooting illumination A data transmission method comprising transmitting at least one of information and scene setting information in the segment unit. Video data to be displayed on a display device, segment delimiter information indicating the start time and / or section of a segment obtained by dividing the video data by an arbitrary number of one or more frames, and the video data in association with each segment Received at least one of camera work information, shooting lighting information, and scene setting information at the time of shooting the video data,
Based on the segment delimiter information and at least one of the video data and / or camera work information, photographing illumination information, and scene setting information, illumination light of an illumination device installed around the display device is controlled. A viewing environment control method characterized by the above.