JP7262317B2 - Remote camera system, control system, control method, and control program - Google Patents

Description

The present disclosure relates to a remote camera system, control system, control method, and control program.

2. Description of the Related Art In recent years, in the field of broadcasting technology, attention has been focused on the technology of a robot camera, in which a camera is attached to the tip of a robot arm to enable photographing by remote control of the robot arm. At filming sites, special equipment such as pedestals and cranes are widely used for the purpose of photographing subjects from various angles and for the purpose of stabilizing the placement of cameras. Skillful manipulation of such special equipment requires the skill of a cameraman. On the other hand, in the case of a robot camera, since the robot arm can be freely moved using a controller, the operation is relatively easy.

Regarding the operation system of a robot camera, Patent Document 1 below proposes a mechanism in which a robot camera model is used as a controller and the robot camera is moved in conjunction with the movement of the model. In addition, Patent Document 2 below proposes a mechanism that uses an operating device that simulates a robot arm as a controller, detects the rotation angle of each joint of the operating device, and moves a robot camera according to the detection results. there is These mechanisms enable intuitive operations because they can be operated with the feeling of moving the robot arm itself by hand.

JP-A-2004-289379 JP-A-07-146501

However, the above mechanism provides the feeling of directly operating the robot arm, and does not intuitively operate the movement of the camera that takes the image. In addition, it is not easy to elaborately manufacture a model of a robot arm or a manipulator that imitates a robot arm.

A robot arm with a complicated structure has a high degree of freedom of movement, and the use of a robot camera has the advantage of facilitating shooting from angles that are difficult to achieve with conventional shooting equipment. Therefore, there is a demand for an operation system that allows the user to operate the robot camera as if it were moving the camera more directly without being conscious of the configuration and movement of the robot arm.

According to one aspect of the present disclosure, an object of the present disclosure is to provide a remote camera system, control system, control method, and control program that enable more intuitive operation.

According to a first aspect of the present disclosure, a first movement that rotates a coordinate system whose origin is a reference point that is connected to the main camera at one end and has a known positional relationship with the one end , and a motion unit capable of a second movement that moves the reference point, and the representative point set on the imaging device of the main camera and the reference point are matched to control the motion unit, A remote camera system may be provided that includes a control system that changes the pose of the main camera with a first movement and changes the position of the main camera with a second movement.

Further, according to the second aspect of the present disclosure, a first coordinate system that rotates a main camera and a reference point that is connected at one end to the main camera and has a known positional relationship with the one end as an origin is a coordinate system. and a motion unit capable of moving the reference point, the unit control unit controls the representative point set on the imaging element of the main camera and the A control system may be provided that aligns the reference points and controls the motion unit to change the pose of the main camera with a first movement and the position of the main camera with a second movement.

Further, according to the third aspect of the present disclosure, the computer rotates a coordinate system having a main camera and a reference point connected at one end to the main camera and having a known positional relationship with the one end as an origin. and a motion unit capable of a second motion to move the reference point, the step of controlling a computer configured on the imaging element of the main camera to A control method can be provided for matching the representative point and the reference point and controlling the motion unit to change the pose of the main camera by the first movement and change the position of the main camera by the second movement. Moreover, according to the fourth aspect of the present disclosure, a control program that causes a computer to execute the processing of the steps can be provided. Further, according to a fifth aspect of the present disclosure, a non-transitory computer readable storage medium storing the control program can be provided.

According to the present disclosure, more intuitive operation becomes possible.

1 is a block diagram for explaining a remote camera system according to this embodiment; FIG. It is a block diagram for explaining the configuration of a camera unit according to the present embodiment. 1 is a diagram schematically showing one configuration example (configuration example 1) of a camera unit according to the present embodiment; FIG. FIG. 3 is a diagram schematically showing one configuration example (configuration example 2) of a camera unit according to the present embodiment; It is a figure which showed typically one structural example (structure example 3) of the camera unit which concerns on this embodiment. It is a figure which showed typically one structural example (structure example 4) of the camera unit which concerns on this embodiment. It is a figure which showed typically one structural example (structure example 5) of the camera unit which concerns on this embodiment. It is a figure which showed typically one structural example (structure example 6) of the camera unit which concerns on this embodiment. FIG. 11 is a diagram schematically showing one configuration example (configuration example 7) of a camera unit according to the present embodiment; It is a block diagram for explaining the function of the control system according to the present embodiment. FIG. 4 is a schematic diagram for explaining location control of a camera unit in the remote camera system according to the embodiment; FIG. 4 is a schematic diagram for explaining TCP settings in the camera unit according to the embodiment; 4 is a chart for explaining TCP setting information according to the embodiment; FIG. (a) is a schematic diagram for explaining tilt control of the camera unit according to the embodiment, and (b) is a schematic diagram for explaining pan control of the camera unit according to the embodiment. FIG. 4 is an explanatory diagram for explaining the relationship among a studio coordinate system, a unit coordinate system, and a camera coordinate system that can define movements of a camera unit and a main camera according to the embodiment; (a) is a schematic diagram for explaining the position (TCP) control of the camera unit according to the present embodiment, and (b) is an operation that can be used for the position (TCP) control of the camera unit according to the present embodiment. It is a schematic diagram for demonstrating a means. (a) is a schematic diagram for explaining the position (TCP) control of the camera unit according to the present embodiment, and (b) is an operation means that can be used for the position (TCP) control of the camera unit according to the present embodiment. It is a schematic diagram for explaining about. (a) is a schematic diagram for explaining the position (TCP) control of the camera unit according to the present embodiment, and (b) is an operation means that can be used for the position (TCP) control of the camera unit according to the present embodiment. It is a schematic diagram for explaining about. FIG. 4 is a schematic diagram for explaining movement control of a camera unit based on a fixed point (eTCP) according to the embodiment; FIG. 3 is a schematic diagram for explaining eye view control according to the present embodiment; FIG. 4 is a schematic diagram for explaining rear view control according to the present embodiment; FIG. 4 is a first chart for explaining preset information according to the embodiment; FIG. FIG. 4 is a second chart for explaining preset information according to the embodiment; FIG. 4 is a chart for explaining scenario information according to the embodiment; 4 is a chart for explaining operation authority information according to the embodiment; It is a block diagram for explaining the function of the operator system according to the present embodiment. FIG. 4 is a diagram schematically showing an example of a controller that can be used as a UI of the operator system according to this embodiment; FIG. 7 is a diagram schematically showing another example of a controller that can be used as a UI of the operator system according to this embodiment; (a) is a diagram schematically showing a mounting form of a gesture operation type device that can be used as the UI of the operator system according to this embodiment, and (b) can be used as the UI of the operator system according to this embodiment. 1 is a diagram schematically showing an operation form of a gesture operation type device; FIG. FIG. 4 is a diagram schematically showing an example of a seated operation unit that can be used as a UI of the operator system according to this embodiment; FIG. 3 is a schematic diagram for explaining the UI (real-time operation mode) of the operator system according to this embodiment; FIG. 4 is a schematic diagram for explaining the UI (sub-view mode) of the operator system according to this embodiment; 4 is a schematic diagram for explaining the UI (preset imaging mode) of the operator system according to the embodiment; FIG. FIG. 4 is a schematic diagram for explaining the UI (scenario shooting mode) of the operator system according to the embodiment; FIG. 5 is a flow diagram for explaining the flow of processing relating to preset registration according to the present embodiment; FIG. 4 is a flow diagram for explaining the flow of processing relating to trace registration according to the embodiment; FIG. 4 is a flow diagram for explaining the flow of processing regarding scenario setting according to the present embodiment; FIG. 5 is a sequence diagram for explaining the flow of processing regarding setting and transfer of operation authority according to the present embodiment; FIG. 3 is a block diagram for explaining hardware capable of realizing the functions of the control system according to the embodiment;

Embodiments of the present disclosure will be described below with reference to the accompanying drawings. Elements having substantially the same functions in the present specification and drawings may be denoted by the same reference numerals, thereby omitting redundant description.

[1. remote camera system]
First, a remote camera system according to this embodiment will be described with reference to FIG. FIG. 1 is a block diagram for explaining the remote camera system according to this embodiment. The remote camera system 5 shown in FIG. 1 is an example of a remote camera system according to this embodiment.

As shown in FIG. 1, the remote camera system 5 includes camera units 10a, 10b, rearview cameras 20a, 20b, operator systems 30a, 30b, and a control system 40. As shown in FIG. For simplicity of explanation, the number of camera units is set to two, the number of rear view cameras to two, and the number of operator systems to two, but any number other than two may be used.

The camera units 10a and 10b are equipment arranged in a studio and used mainly for shooting video for broadcasting. As will be described later, the main configurations of the camera units 10a and 10b are the same. The camera units 10a and 10b may include a damping mechanism for suppressing vibration, and may also include a silent mechanism for suppressing operation noise. As a vibration damping mechanism, for example, a system that uses elastic members such as rubber or springs to dampen vibrations, or a damping system that suppresses vibrations by moving the housing in the opposite phase according to the output of a motion sensor such as a gyro sensor. A vibration system or the like can be applied. Silent mechanisms include sound absorbing members arranged inside and outside the housing. Quietness may also be improved by using a cooling fan that lowers the temperature inside the housing, a power supply fan mounted on the power supply, or the like to be a highly silent fan.

The rear-view cameras 20a and 20b are cameras that are arranged in a studio and can be used to shoot images from the surroundings of the camera units 10a and 10b at viewpoints including the camera units 10a and 10b in the angle of view. For convenience of explanation, the rear view cameras 20a and 20b are installed behind the camera units 10a and 10b, and an image from a viewpoint (hereinafter referred to as a rear view) including the camera units 10a and 10b in an angle of view (hereinafter referred to as a rear view) is described below. video) will be described as an example, but the arrangement of the rear view cameras 20a and 20b is not limited to this example. Although referred to as a "rear view" camera, it may be a camera that takes pictures of the camera units 10a, 10b from the side, obliquely forward, above, or below. As will be described later, the main configurations of the rearview cameras 20a and 20b are the same.

The operator systems 30a and 30b are systems used by operators who operate the camera units 10a and 10b. The operator systems 30a, 30b may also be used to operate the rear view cameras 20a, 20b. For example, the operator may use one of the operator systems 30a, 30b or another operator system to operate the pan/tilt of the rear view cameras 20a, 20b. As will be described later, the main configuration of operator systems 30a and 30b is the same. In the example of FIG. 1, the operator system 30a is located in the studio, and the operator system 30b is located in the control room (eg, sub-control room, relay car, etc.). Thus, operator systems 30a and 30b may be used in a studio, in a control room, or in other locations (eg, another studio or a remote broadcast station). etc.).

The control system 40 is a system that controls the operations of the camera units 10a and 10b according to operation inputs from the operator systems 30a and 30b. In addition, the control system 40 has a function of providing the operator systems 30a and 30b with rear-view images output from the rear-view cameras 20a and 20b, and the operation authority of the camera units 10a and 10b set for the operator systems 30a and 30b. It has functions such as management. In the example of FIG. 1, control system 40 is located in a control room.

The system configuration in FIG. 1 will be described below as an example, but the system configuration of the remote camera system according to this embodiment is not limited to the example in FIG. For example, the remote camera system according to this embodiment is not limited to studio photography, but can also be applied to photography at indoor and outdoor sports facilities, event venues, concert halls, convention venues, ceremony halls, and the like. Each component of the remote camera system 5 will be further described below.

[2. camera unit]
The configuration of the camera unit according to this embodiment will be described with reference to FIG. FIG. 2 is a block diagram for explaining the configuration of the camera unit according to this embodiment. As described above, since the main configurations of the camera units 10a and 10b are the same, the camera unit 10a will be described in detail, and the description of the camera unit 10b will be omitted.

As shown in FIG. 2, the camera unit 10a has a main camera 11, an eye view camera 12, a motion unit 13, and a control box .

The main camera 11 is an imaging device for outputting broadcast video. The main camera 11 includes optical elements such as lenses and optical filters, imaging elements such as CCDs (Charge-Coupled Devices) and CMOSs (Complementary Metal-Oxide-Semiconductors), DSPs (Digital Signal Processors), and CPUs (Central Processing Units). , a GPU (Graphic Processing Unit) or the like.

The main camera 11 also has a communication interface for communicating with the control system 40 through a wired or wireless network or a dedicated communication line. An image output from the main camera 11 (hereinafter referred to as main image) is transmitted to the control system 40 via a communication interface. Note that the main camera 11 may be equipped with a microphone. Also, the main camera 11 may be equipped with communication means for direct communication with the operator systems 30a and 30b.

The eye-view camera 12 is an imaging device for providing the operator who operates the camera unit 10a with an image of the inside of the studio seen from the cameraman's point of view. The eye-view camera 12 is preferably installed at or near the position where the cameraman's eyes would be positioned if the main camera 11 were to be operated manually by the cameraman. For example, the eye-view camera 12 may be physically fixed to the housing of the main camera 11 .

The eye-view camera 12 is a wide-angle camera having an angle of view wider than the human field of view. For example, the eye view camera 12 may be an omnidirectional camera capable of obtaining an omnidirectional video (360° video), or a wide range of video (180° video, 270° video, etc.) surrounding the camera unit 10a. ) may be used. The eye view camera 12 is equipped with a communication interface for communicating with the control system 40 through a wired or wireless network or a dedicated communication line. Images output from the eye view camera 12 are transmitted to the control system 40 via the communication interface. Note that the eye view camera 12 may be equipped with communication means for direct communication with the operator systems 30a and 30b.

The motion unit 13 is equipment for changing the position (hereinafter referred to as location) of the camera unit 10a in the studio and the reference position (hereinafter referred to as position) and orientation of the main camera 11 . The motion unit 13 may take many different forms. For example, the location may be fixed, or the position and attitude change range may be restricted. Here, possible forms of the motion unit 13 (configuration example 1 to configuration example 7 of the camera unit 10a) will be described with reference to FIGS. 3 to 9. FIG.

A configuration example (configuration example 1) of the camera unit according to the present embodiment will be described with reference to FIG. FIG. 3 is a diagram schematically showing one configuration example (configuration example 1) of the camera unit according to the present embodiment.

Configuration Example 1 is a configuration example of the camera unit 10a when the motion unit 13 is configured using the robot arm 13a and the base 13b.

In configuration example 1, the robot arm 13a is a 6-axis robot arm, and can move the position of the main camera 11 to any position in space within the reach of the tip of the arm. Further, the robot arm 13a can tilt the orientation of the main camera 11 in the pan direction and the tilt direction by rotating each joint. The base 13b has a movement mechanism (not shown) such as wheels and a motor, and can move in the studio by self-running. According to Configuration Example 1, the motion unit 13 can freely control the location, position, and pan/tilt.

Next, a configuration example (configuration example 2) of the camera unit according to the present embodiment will be described with reference to FIG. FIG. 4 is a diagram schematically showing one configuration example (configuration example 2) of the camera unit according to the present embodiment.

Configuration example 2 is a configuration example of the camera unit 10a when the motion unit 13 is configured using the robot arm 13a, the lifting mechanism UD1, and the base 13b1.

In configuration example 2, the robot arm 13a is a 6-axis robot arm, and can move the position of the main camera 11 to any position in space within the reach of the tip of the arm. Further, the robot arm 13a can tilt the direction of the main camera 11 in the pan direction and the tilt direction by rotating each joint. The lifting mechanism UD1 can lift and lower the robot arm 13a. The base 13b1 has wheels for moving along rails, and can move within the studio along a moving path defined by the rails. According to Configuration Example 2, the motion unit 13 can change its location along the rail and freely control its position and pan/tilt.

Next, a configuration example (configuration example 3) of the camera unit according to the present embodiment will be described with reference to FIG. FIG. 5 is a diagram schematically showing one configuration example (configuration example 3) of the camera unit according to the present embodiment.

Configuration example 3 is a configuration example of the camera unit 10a when the motion unit 13 is configured using the robot arm 13a, the pedestal UD2, and the base 13b.

In configuration example 3, the robot arm 13a is a 6-axis robot arm, and can move the position of the main camera 11 to any position in space within the reach of the tip of the arm. Further, the robot arm 13a can tilt the orientation of the main camera 11 in the pan direction and the tilt direction by rotating each joint. The pedestal UD2 can raise and lower the robot arm 13a. The base 13b has a movement mechanism (not shown) such as wheels and a motor, and can move in the studio by self-running. According to Configuration Example 3, the motion unit 13 can freely control the location, position, and pan/tilt.

Next, a configuration example (configuration example 4) of the camera unit according to the present embodiment will be described with reference to FIG. FIG. 6 is a diagram schematically showing one configuration example (configuration example 4) of the camera unit according to the present embodiment.

Configuration example 4 is a configuration example of the camera unit 10a when the motion unit 13 is configured using the robot arm 13a and the tripod 13b2.

In configuration example 4, the robot arm 13a is a six-axis robot arm, and can move the position of the main camera 11 to any position in space within the reach of the tip of the arm. Further, the robot arm 13a can tilt the orientation of the main camera 11 in the pan direction and the tilt direction by rotating each joint. The tripod 13b2 is immobile and supports the robot arm 13a at a specific location. According to Configuration Example 4, the motion unit 13 cannot change the location, but can freely control the position and pan/tilt at a specific location.

Next, one configuration example (configuration example 5) of the camera unit according to the present embodiment will be described with reference to FIG. FIG. 7 is a diagram schematically showing one configuration example (configuration example 5) of the camera unit according to the present embodiment.

Configuration example 5 is a configuration example of the camera unit 10a when the motion unit 13 is configured using the platform 13a1, the lifting mechanism UD3, and the base 13b.

The camera platform 13a1 can tilt the direction of the main camera 11 in the pan direction and the tilt direction by rotating the rotation shaft. The elevating mechanism UD3 can elevate the camera platform 13a1. The base 13b has a movement mechanism (not shown) such as wheels and a motor, and can move in the studio by self-running. According to Configuration Example 5, the motion unit 13 can freely change its location, can change its position in the vertical direction, and can freely control panning and tilting.

Next, a configuration example (configuration example 6) of the camera unit according to the present embodiment will be described with reference to FIG. FIG. 8 is a diagram schematically showing one configuration example (configuration example 6) of the camera unit according to the present embodiment.

Configuration example 6 is a configuration example of the camera unit 10a when the motion unit 13 is configured using the crane 13a2, the pedestal UD4, and the base 13b.

The crane 13a2 can move the position of the main camera 11 in the horizontal plane with the connecting portion to the pedestal UD4 as a base point, and tilts the main camera 11 in the tilt direction while lowering the position thereof by tilting the arm of the pedestal UD4. be able to. Also, the direction of the main camera 11 can be tilted in the pan direction and the tilt direction by a rotation mechanism provided at the tip of the crane 13a2. The pedestal UD4 can raise and lower the crane 13a2. The base 13b has a movement mechanism (not shown) such as wheels and a motor, and can move in the studio by self-running. According to Configuration Example 6, the motion unit 13 can freely control the location, and can control the position and pan/tilt within the movable range of the crane 13a2.

Next, a configuration example (configuration example 7) of the camera unit according to the present embodiment will be described with reference to FIG. FIG. 9 is a diagram schematically showing one configuration example (configuration example 7) of the camera unit according to the present embodiment.

Configuration example 7 is a configuration example of the camera unit 10a when the motion unit 13 is configured using the robot arm 13a3 and the base 13b.

The robot arm 13a3 is a 7-axis robot arm having a tip rotation mechanism and axes a to f, and can move the position of the main camera 11 to any position in space within the reach of the tip of the arm. . Further, the robot arm 13a can tilt the direction of the main camera 11 in the pan direction and the tilt direction by rotating each joint. The base 13b has a movement mechanism (not shown) such as wheels and a motor, and can move in the studio by self-running. According to Configuration Example 7, the motion unit 13 can freely control the location, position, and pan/tilt.

As described above, the motion unit 13 can take various forms, but for the sake of simplicity, the following explanation will be given by taking the configuration example 1 shown in FIG. 3 as an example.

Refer to FIG. 2 again. The motion unit 13 has a robot arm 13a and a base 13b. Motion unit 13 is connected to control box 14 and moves robot arm 13 a and base 13 b in response to control signals received from control system 40 via control box 14 . Although not shown, the control box 14 may include a communication interface for communicating with the control system 40, a processor for controlling the camera unit 10a, and a power supply for driving the camera unit 10a.

[3. control system]
Next, referring to FIG. 10, the functions of the control system according to the present embodiment will be described using the control system 40 as an example. FIG. 10 is a block diagram for explaining the functions of the control system according to this embodiment.

As shown in FIG. 10, the control system 40 includes a unit control section 41, a main video output section 42, an eye view providing section 43, a rear view providing section 44, a communication tool providing section 45, and a camerawork execution section 46. , an authority management unit 47 , and a storage unit 48 .

The unit control section 41 controls operations of the camera units 10a and 10b. For example, the unit control section 41 controls the imaging operation of the main camera 11, the imaging operation of the eye-view camera 12, the position and pan/tilt control of the robot arm 13a, and the location control of the base 13b. .

(About location control)
Here, location control of the camera unit in the remote camera system according to this embodiment will be described with reference to FIG. 11 . FIG. 11 is a schematic diagram for explaining location control of the camera unit in the remote camera system according to this embodiment.

FIG. 11 schematically shows the inside of a studio in which performers m1, . In the following description, for simplicity of notation, the camera unit 10a may be denoted by UN1, the camera unit 10b by UN2, the rear-view camera 20a by RVC1, the rear-view camera 20b by RVC2, and the main camera 11 by MC.

In the example of FIG. 11, five locations (LOC1, . . . , LOC5) are set in the studio. LOC1 and LOC2 are locations set in the camera warehouse. For example, the unit control section 41 controls the base 13b to move UN1 from LOC1 to LOC3 along the movement route MV13 before the start of imaging. Similarly, the unit control section 41 moves UN2 from LOC2 to LOC5 along the movement path MV25 before the start of imaging.

The positions of UN1 and UN2 can be represented by a two-dimensional coordinate system whose origin is the reference point s0 set in the studio. In the following description, to simplify the description, a coordinate system defined by two orthogonal axes sX and sY with s0 as the origin is assumed, and the coordinate values of this coordinate system are used to set in the studio. , the location of UN1, UN2 within the studio, and the location of RVC1, RVC2. For example, LOCn (n=1, . . . , 5) can be represented by coordinate values (sXn, sYn).

As a method of specifying the location of UNk (k=1, 2) at a certain point in time, for example, there is a method of placing a station that transmits radio waves in a studio and specifying it based on the strength of the radio waves received by UNk. Another method is to equip the UNk with a GPS (Global Positioning System) unit and use GPS data to identify the location of the UNk. Furthermore, there is a method of mounting a motion sensor on UNk, obtaining a moving route based on the output of the motion sensor, and identifying the current location. There is also a method of specifying the current location by video analysis of rear-view video including images of UNk using the rear-view cameras 20a and 20b. As a video analysis method, for example, object recognition or tracking technology can be applied. Also, AI (Artificial Intelligence) technology may be applied to improve the accuracy of video analysis. Note that the location specifying method is not limited to these examples, and any method can be applied.

The unit control section 41 grasps the location of UNk by the method described above, and controls the location of UNk according to the operation of the operator. By being able to grasp the locations of UN1 and UN2 in real time, the unit control section 41 can shift the movement routes of both spatially and temporally so that the movement routes of UN1 and UN2 do not intersect, thereby avoiding collision between the two. , UN2 can avoid including the main image, or avoid including objects that should not be included in the main image (eg, rails, lighting, etc.). The location control may be executed by an operator's real-time operation, or may be executed based on control contents (presets and scenarios described later) set in advance by the operator.

(Regarding TCP (Tool Center Point) settings)
Next, TCP settings in the camera unit according to this embodiment will be described with reference to FIGS. FIG. 12A is a schematic diagram for explaining TCP settings in the camera unit according to this embodiment. FIG. 12B is a schematic diagram for explaining TCP settings in the camera unit according to this embodiment.

A control box 14 controls the movement of each joint (rotation angle of each axis) of the robot arm 13a. A reference point called TCP is set on the robot arm 13a. For example, the TCP is set near the tip of the robot arm 13a (the end to which the main camera 11 is connected). The control box 14 specifies the rotation angle of each axis corresponding to the displacement according to the control signal that conveys the displacement of the TCP, and rotates each axis of the robot arm 13a by the specified rotation angle. In other words, if the position of the TCP is designated, the control box 14 automatically transforms the robot arm 13a into an appropriate shape.

The position of TCP can be changed. In the remote camera system 5, the position of the TCP is set so that the representative point set on the imaging element of the main camera 11 (MC) and the TCP of the robot arm 13a match.

For example, as shown in FIG. 12(a), the TCP (TCP0) preset in the robot arm 13a is changed to the point TCP1 on the image sensor IS1 of MC. TCP1 is the intersection of the image sensor IS1 and the optical axis AX of the lens system LS1. By setting the TCP of the robot arm 13a to TCP1, the operator can move the MC position based on TCP1. In other words, the operator moves the MC with reference to the image plane of the MC, which provides a more intuitive feeling of operation.

Normally, a connecting portion (for example, a screw hole) provided on the bottom surface of the casing of the MC is provided directly above the optical axis AX. Also, the distance FB1 from the flange surface to the image sensor IS1 is known. Therefore, the distance d11 on the optical axis AX between TCP0 and TCP1 can be easily obtained. Also, since the optical axis AX passes through the center of the lens system LS1, the distance d12 between the optical axis AX and TCP0 can be easily obtained. As shown in FIG. 12(b), even if the type of MC changes, the positional relationship between TCP0 and the changed TCP (TCP2) can be easily obtained by the same method.

Information indicating the positional relationship between TCP0 and the changed TCP in the robot arm 13a may be stored in advance in the storage unit 48 as the TCP setting information 48a. The TCP setting information 48a includes, for example, information as shown in FIG. FIG. 13 is a chart for explaining TCP setting information according to this embodiment.

As shown in FIGS. 12A and 12B, considering a coordinate system in which the optical axis AX is the cY axis, the row direction of the image sensor is the cZ axis, and the column direction of the image sensor is the cX axis, TCP0 and The amount of TCP movement, which indicates the positional relationship between , and TCP, can be expressed mainly by the displacement dcY along the cY axis and the displacement dcZ along the cZ axis. If TCP0 is not directly under the optical axis AX, then the TCP displacement includes the displacement dcX along the cX axis.

The unit control unit 41 refers to the TCP setting information 48a, identifies the TCP movement amount corresponding to the type of MC included in UNk, and transmits the identified TCP movement amount to the control box 14 of UNk to change the position of TCP. do. Thereby, even if the type of MC is changed, the operator can move the position of MC with reference to the image plane of MC. In other words, the operator can intuitively control the position of the MC without being conscious of the type of MC and without being conscious of the movement of the robot arm 13a.

Further, the robot arm 13a can change the pan/tilt of the MC based on the changed TCP, as shown in FIGS. 14(a) and 14(b). FIG. 14A is a schematic diagram for explaining tilt control of the camera unit according to this embodiment. FIG. 14B is a schematic diagram for explaining pan control of the camera unit according to this embodiment. Hereinafter, TCP after change may be simply referred to as TCP. Also, the rotation angle in the tilt direction with the cX axis as the rotation axis is denoted by θx, and the rotation angle in the pan direction with the cZ axis as the rotation axis is denoted by θz.

As shown in FIGS. 14A and 14B, when the MC is panned and tilted with reference to the TCP set at the intersection of the optical axis and the image sensor, the image plane is tilted on the spot. Therefore, the operator can intuitively control the panning and tilting of the MC without being conscious of the type of MC and without being conscious of the movement of the robot arm 13a.

(Arrangement of coordinate system)
Here, with reference to FIG. 15, the relationships among the studio coordinate system, the unit coordinate system, and the camera coordinate system that define the movements of the camera unit and the main camera according to this embodiment will be summarized. FIG. 15 is an explanatory diagram for explaining the relationship among the studio coordinate system, the unit coordinate system, and the camera coordinate system that can define the movement of the camera unit according to this embodiment.

Hereinafter, the coordinate system defined by the sX-axis and sY-axis described above will be referred to as a "studio coordinate system". Also, the coordinate system defined by the cX-axis, cY-axis, and cZ-axis described above will be called a "camera coordinate system". The origin Oc of the camera coordinate system is the position of the TCP. Further, the coordinate point indicating the location of UNk (k = 1, 2) is the origin, the axis orthogonal to the studio coordinate system is the uZ axis, one axis orthogonal to the uZ axis is the uX axis, and the uZ axis and the uX axis are orthogonal. Introduce a “unit coordinate system” in which the uY axis is the other axis that Let the origin Ou of the unit coordinate system for UNk be the location of UNk.

As shown in FIG. 15, when the unit coordinate system is introduced, the positional relationship between the location of UNk and TCP can be represented by the coordinate points (uX, uY, uZ) of the unit coordinate system. That is, the movement of TCP can be expressed as the movement of TCP in the unit coordinate system. Also, MC pan/tilt control based on TCP can be expressed as movement in the optical axis direction in the camera coordinate system. The optical axis direction can be expressed, for example, by a unit vector extending from the origin Oc of the camera coordinate system to the object side along the optical axis.

A change in the location of UNk is expressed as a change in location coordinates in the studio coordinate system. is designated to control the TCP of the MC, and a coordinate point of the camera coordinate system is designated to control the pan/tilt of the MC. In this way, by controlling the position and attitude of MC based on one point, TCP, it is possible to point the camera in the same direction from the same position with the same mechanism even if the structure of UNk changes.

(Regarding three-sided independent control)
As described above, by applying the motion unit 13, the TCP can be moved three-dimensionally and the position of the MC can be freely moved. However, from the viewpoint of making it easier for the operator to grasp the movement of the MC, the remote camera system 5 may be provided with a control mechanism (hereinafter referred to as three-plane independent control) as shown in FIGS. 16(a) to 18(b). good. This mechanism consists of a first operation (FIGS. 16(a) and (b)) for moving the TCP within the uX-uZ plane, and a second operation (FIGS. 17(a) and 17(a) for moving the TCP within the uY-uZ plane. (b)) and the third operation (FIGS. 18(a) and (b)) of moving in the uX-uY plane. The first operation, the second operation, and the third operation will be described below in order.

FIG. 16(a) schematically shows how the MC moves so as to pass through four positions (POS11, . . . , POS14) in the uX-uZ plane. FIG. 16(b) also shows an operation interface 301 and an operation body 302 for operating a first operation area (uZ-uX OPERATION) of the operation interface 301, which is used for the first operation. . FIG. 16A is a schematic diagram for explaining the position (TCP) control of the camera unit according to this embodiment. FIG. 16B is a schematic diagram for explaining operation means that can be used for position (TCP) control of the camera unit according to this embodiment.

The first operation area corresponds to the uZ-uX plane, and as shown in FIG. 16(b), when the operation body 302 operates the first operation area, as shown in FIG. TCP moves accordingly. In the example of FIG. 16(b), since the operating body 302 moves in a counterclockwise circle, the TCP also moves counterclockwise in the uZ-uX plane following this movement. Thus, by making the first operation area independent, the movement of MC can be restricted within the uZ-uX plane. By restricting the movement of the MC within the uZ-uX plane, the operator can perform the operation without anxiety while recognizing that the MC is moving within the plane perpendicular to the optical axis.

FIG. 17(a) schematically shows how MC moves so as to pass through four positions (POS21, . . . , POS24) in the uY-uZ plane. FIG. 17(b) also shows an operation interface 301 and an operation body 302 for operating a second operation area (uY-uZ OPERATION) of the operation interface 301, which is used for the second operation. . FIG. 17A is a schematic diagram for explaining the position (TCP) control of the camera unit according to this embodiment. FIG. 17B is a schematic diagram for explaining operating means that can be used for position (TCP) control of the camera unit according to this embodiment.

The second operation area corresponds to the uYuZ plane, and as shown in FIG. 17(b), when the operation body 302 operates the second operation area, as shown in FIG. TCP moves accordingly. In the example of FIG. 17(b), since the operating body 302 moves in a counterclockwise circle, the TCP also moves counterclockwise in the uY-uZ plane following this movement. Thus, by making the second operation area independent, the movement of MC can be restricted within the uY-uZ plane. By restricting the movement of the MC within the uY-uZ plane, the operator can safely perform operations while recognizing that the MC is moving within the vertical plane including the optical axis.

FIG. 18(a) schematically shows how MC moves so as to pass through four positions (POS31, . . . , POS34) in the uX-uY plane. FIG. 18(b) also shows an operation interface 301 and an operation body 302 for operating a third operation area (uX-uY OPERATION) of the operation interface 301, which is used for the third operation. . FIG. 18A is a schematic diagram for explaining the position (TCP) control of the camera unit according to this embodiment. FIG. 18B is a schematic diagram for explaining operating means that can be used for position (TCP) control of the camera unit according to this embodiment.

The third operation area corresponds to the uX-uY plane, and as shown in FIG. 18(b), when the operation body 302 operates the third operation area, as shown in FIG. TCP moves accordingly. In the example of FIG. 18(b), since the operating body 302 moves in a counterclockwise circle, the TCP also moves counterclockwise within the uX-uY plane following this movement. Thus, by making the third operation area independent, the movement of MC can be restricted within the uX-uY plane. By restricting the movement of the MC within the uX-uY plane, the operator can perform operations without worry while recognizing that the MC is moving within the horizontal plane including the optical axis.

The operation interface 301 may be a GUI (Graphical User Interface) displayed on the operator systems 30a and 30b, or may be an operation system of a controller to be described later. When the first to third operations are independently assigned to the operation system of the controller, unlike the operation areas shown in FIGS. First to third operations are assigned to the means. An operation system that provides the functions of the operation interface 301 will be further described later.

(Modified example of three-plane independent control)
In the above description, an operation system that independently controls movements within three planes formed by two of the three axes that define the unit coordinate system has been described. It is also possible to realize a similar operation system by replacing with In this case, by replacing uX with cX, uY with cY, and uZ with cZ, three-plane independent control based on the camera coordinate system can be realized.

Furthermore, a first plane perpendicular to the optical axis, a second plane including an axis indicating the upward direction of the main camera (for example, the column direction of the image sensor) and the optical axis, and another plane perpendicular to the axis and the optical axis Assuming a third plane containing the axis and the optical axis, an operation system that independently controls movement within the first plane, movement within the second plane, and movement within the third plane. Realization is also possible. For example, if the uZ-uX plane is replaced with the first plane, the uY-uZ plane with the second plane, and the uX-uY plane with the third plane, the three-plane independent control mechanism described above can be applied as it is. can. These modifications also naturally belong to the technical scope of the present embodiment.

(Regarding movement control based on fixed points)
Next, movement control of the camera unit based on a fixed point (eTCP) according to this embodiment will be described with reference to FIG. FIG. 19 is a schematic diagram for explaining movement control of the camera unit based on a fixed point (eTCP) according to this embodiment.

As a method of restricting the movement of MC, as shown in FIG. 19, a fixed point (eTCP) as a target is set, the optical axis passes through eTCP, and TCP is controlled to draw a circular orbit centered on eTCP. There is a way. When this method is applied, the unit control section 41 moves the origin Oc of the camera coordinate system along the above-described circular orbit, and adjusts the pan angle in the camera coordinate system so that the unit vector indicating the optical axis points to eTCP. Control θz. In the example of FIG. 19, for convenience of explanation, the MC is shown moving two-dimensionally, but it can also be moved freely on a spherical surface centered on eTCP. In this case, not only the pan angle θz in the camera coordinate system but also the tilt angle θx can be controlled. According to this control method, the object located at the eTCP position can be easily photographed from various angles.

As described above, the unit control section 41 controls the locations of the camera units 10a and 10b, moves the TCP to control the position of the main camera 11, and rotates the main camera 11 with the axis passing through the TCP as the rotation axis. Pan/tilt control. With this mechanism, a more intuitive operation system can be realized. In addition, since the position and orientation of the main camera 11 can be specified by coordinate values in the studio coordinate system, unit coordinate system, and camera coordinate system, it is possible to reproduce a specific shooting situation by specifying these coordinate values. become.

Refer to FIG. 10 again. The main video output unit 42 processes the video signal of the main video output from the main camera 11 and provides the main video to the operator systems 30a and 30b. For example, the main video output unit 42 may perform down-conversion processing to lower the resolution of the main video or compression processing of the video signal, and provide the processed main video to the operator systems 30a and 30b. These processes can reduce the load when transmitting the main video from the control system 40 to the operator systems 30a and 30b. Also, the load on the main video display processing in the operator systems 30a and 30b can be reduced.

(About eye view image)
The eye-view providing unit 43 processes the image output from the eye-view camera 12 and provides the processed image as an eye-view image to the operator systems 30a and 30b. For example, the eye-view providing unit 43 acquires information indicating the line-of-sight direction of the operator who is operating the camera unit 10a using the operator system 30a, and determines the range corresponding to the line-of-sight direction in the output image of the eye-view camera 12. (hereinafter referred to as line-of-sight range) is cut out. Then, the eye-view providing unit 43 provides the cut-out image as an eye-view image to the operator system 30a.

When the output image of the eye-view camera 12 is an omnidirectional image, the eye-view providing unit 43 cuts out the eye-view image by the method shown in FIG. FIG. 20 is a schematic diagram for explaining eye view control according to the present embodiment.

As shown in FIG. 20, the omnidirectional image can be regarded as an image projected onto a celestial sphere 43a (virtual spherical surface) centered on the position of the eye view camera 12. FIG. A coordinate system defined by mutually orthogonal eX, eY, and eZ axes is set with the center of the celestial sphere 43a as the origin. , the line-of-sight direction can be expressed by a unit vector pointing to one point (line-of-sight 43c) on the celestial sphere 43a. In addition, in the example of FIG. 20, the eY axis is set to coincide with the optical axis AX.

The human visual field is approximately 60 degrees upward, approximately 70 degrees downward, and approximately 100 degrees toward the ear. A range of 70°, down (towards -eZ) about 60°, right/left (towards +eX/-eX) about 100° is visible. Therefore, the angles L and H defining the field of view range 43b may be set to L=130° and H=200° in accordance with the human visual field. Also, H may be set in the range of 60° to 90°, which corresponds to the stable field of fixation in the horizontal field of view, and L may be set in the range of 45° to 70°, which corresponds to the stable field of fixation in the vertical field of view.

When the operator wears a head-mounted device such as HMD (Head Mounted Display) or smart glasses and the device is equipped with a motion sensor, the operator's line of sight direction is determined based on the output of the motion sensor. can be estimated. In addition, the direction of the operator's line of sight can be estimated by photographing the face of the operator from the front and detecting the line of sight by image processing. By applying a mechanism in which these estimation results are input to the control system 40 in real time, the visual field range 43b can be made to follow the line of sight of the operator.

Since the eye-view camera 12 is installed near the location where the cameraman's line of sight is located, the operator can actually operate the main camera 11 in the studio by providing the operator with an image obtained by clipping the field of view 43b. You can get the view of the virtual cameraman who is there.

(About rear view video)
The rearview providing unit 44 processes video signals of the rearview images output from the rearview cameras 20a and 20b and provides the rearview images to the operator systems 30a and 30b. In addition, the rear-view providing unit 44 may control pan/tilt/zoom of the rear-view cameras 20a and 20b according to the operator's operation.

The rear-view cameras 20a and 20b are cameras that photograph the camera units 10a and 10b from behind and provide rear-view images so that the operator can check the states of the camera units 10a and 10b. Therefore, even if the operator is provided with a rear-view image that does not show the camera unit being operated by the operator, the operator cannot grasp the state of the camera unit from the rear-view image. Therefore, the rear-view providing section 44 may use the relationship between the locations of the camera units 10a and 10b and the locations of the rear-view cameras 20a and 20b to determine which rear-view video to provide to which operator.

Here, with reference to FIG. 11, a method for determining the destination of the rear-view video will be described using the situation shown in FIG. 11 as an example. In the example of FIG. 11, RVC1 and RVC2 are installed in the studio. In addition, UN1 is arranged in LOC3, and UN2 is arranged in LOC5. RVC1 is located behind LOC3, and as shown in FIG. 21, UN1 at LOC3 can be imaged from behind. FIG. 21 is a schematic diagram for explaining rear view control according to the present embodiment. However, UN2 at LOC5 does not fall within the angle of view of RVC1. RVC2 is positioned behind LOC5, and UN1 in LOC3 does not enter the angle of view of RVC2.

For example, when UN1 is operated by the operator system 30a and UN2 is operated by the operator system 30b, the rear view providing unit 44 provides the rear view image output from the RVC1 to the operator system 30a, and the rear view image output from the RVC2. The video is provided to the operator system 30b. If both RVC1 and RVC2 are in a situation where they can capture the UN at LOC4, the rear view providing unit 44 displays the output of the RVC that displays the entire UN or displays the UN in a larger size. The operator may be provided with the output of the RVC that is present.

(About communication tools)
Refer to FIG. 10 again. The communication tool providing unit 45 provides a communication tool for communicating by voice or text between the operators who operate the camera units 10a and 10b, or between the operator and other staff involved in the imaging work. For example, the communication tool providing unit 45 provides a chat tool for exchanging character information between the operator systems 30a and 30b. Further, the communication tool providing unit 45 may store character information exchanged between the operator systems 30a and 30b in the storage unit 48 as a communication log 48f.

(Regarding presets, scenarios, and real-time operations)
The camerawork execution section 46 instructs the unit control section 41 to perform camerawork in response to operations by the operator systems 30a and 30b, and controls the camera units 10a and 10b via the unit control section 41. FIG. The camerawork execution unit 46 has a preset execution unit 46a, a scenario execution unit 46b, and a real-time operation unit 46c.

The preset executing section 46 a controls the operations of the camera units 10 a and 10 b based on the preset DB 48 b including location, position, pan/tilt, and setting information of the main camera 11 . The preset DB 48b includes contents as shown in FIGS. 22 and 23. FIG. FIG. 22 is a first chart for explaining preset information according to this embodiment. FIG. 23 is a second chart for explaining the preset information according to this embodiment.

As shown in FIG. 22, the preset DB 48b includes a preset number (PRESET NUMBER), location information (LOCATION), main camera positioning information (POSITIONING), zoom information, and thumbnails. .

The preset number is an identification number for identifying individual preset information. The location information includes information on the location where the camera unit is arranged. For example, the location information may include identification information for identifying the location and coordinate values of the studio coordinate system corresponding to the location.

The positioning information of the main camera includes TCP coordinate information. TCP can be represented by coordinate values of the corresponding unit coordinate system. The pan/tilt angle can be expressed by a rotation angle θx about the cX axis of the camera coordinate system with TCP as the origin, and a rotation angle θz about the cZ axis. The zoom information may be expressed by the focal length of the lens system, or may be expressed by a combination of the reference focal length and the zoom magnification.

A thumbnail is a thumbnail image generated from a main video. For example, in the preset information with preset number PS001, the camera unit is at LOC4, the TCP is at the position of (10.0, 50.0, 1500) in the unit coordinate system, and the tilt angle is 0.0 degrees. , the thumbnail of the main video output from the main camera 11 of the camera unit when the pan angle is 10.0 degrees and the focal length of the lens system is 50.3 mm.

By controlling the camera unit using the above preset information, the main camera 11 can be directed in the same position and in the same direction as when the main video corresponding to the thumbnail was shot, and the angle of view is also controlled in the same way. , you can get the main image with the same shooting range as the thumbnail. That is, the photographing conditions are almost reproduced with respect to the camera unit. Also, as shown in FIG. 23, focus (FOCUS), iris (IRIS), gain (GAIN), pedestal level (PEDESTAL LEVEL), gamma value (GAMMA), knee (KNEE), color temperature (COLOR TEMP), etc. By taking these parameters into consideration, it is possible to more faithfully reproduce the imaging conditions even if these parameters change.

Note that the focus indicates the in-focus position, the iris indicates the aperture value (e.g., F-number), the gain indicates the degree of amplification of the electrical signal output from the image pickup device, and the pedestal level indicates the luminance that is the reference for the video signal level. The gamma value indicates the response characteristics of the image gradation, the knee is a parameter used when compressing the high luminance side of the video signal that exceeds the dynamic range, and the knee point (P) is the level from which compression is performed. Knee slope (S) indicates how much to compress, and color temperature indicates the set value of white balance. As other parameters, preset parameters related to scenes and frames may be included in the preset DB 48b.

The preset execution section 46a can instruct the unit control section 41 on the preset information selected by the operator from the preset DB 48b, and can control the camera units 10a and 10b via the unit control section 41. FIG. The operator can easily operate the camera unit simply by appropriately selecting preset information from the preset DB 48b.

Hereinafter, an operation mode in which the operator can proceed with imaging by selecting preset information may be referred to as a preset imaging mode. In the preset photographing mode, the preset executing section 46a executes control of the camera unit based on the preset information in response to the operator's selection operation of the preset information. At this time, while the camera unit is being moved to the location indicated by the preset information, the preset execution unit 46a controls at least some of various parameters such as the position of the main camera, pan/tilt, zoom, and focus based on the preset information. You may As a result, the control can be completed in a shorter period of time than when the camera unit is moved to a specific location and then the position control is performed. Operation of the camera unit based on the preset DB 48b will be further described later.

Refer to FIG. 10 again. The scenario executing section 46b controls the operations of the camera units 10a and 10b based on a scenario DB 48c that includes scenarios in which preset information contained in the preset DB 48b is arranged in chronological order. The scenario DB 48c includes contents as shown in FIG. FIG. 24 is a chart for explaining scenario information according to this embodiment.

As shown in FIG. 24, the scenario DB 48c includes a timeline (TIME) and preset information of the preset DB 48b used for controlling each camera unit (UN1, UN2). Note that the scenario DB 48c may include a plurality of scenarios (SCN1, SCN2, and SCN3 in the example of FIG. 24).

In the example of SCN1, UN1 starts operating based on PS001, and performs operations based on preset information in the order of PS002, PS003, and PS006. The operating time of UN1 is defined for each preset. In the example of FIG. 24, for convenience of explanation, it is described as if each time on the timeline is associated with the preset information, but this is a schematic representation of a rough correspondence on the timeline. Yes, and as described above, the scenario DB 48c contains information indicating from what time to what time the preset is executed. In this way, UN1 and UN2 perform actions based on the scenario information set in the scenario DB 48c at a certain time. The scenario executing section 46b performs state control of UN1 and UN2 based on the preset DB 48b.

In the case of an operation using the scenario DB 48c, the operator simply selects a scenario in the scenario DB 48c, the scenario executing section 46b controls UN1 and UN2 via the unit control section 41, and the main video is output from UN1 and UN2. be. In other words, the scenario executing section 46b realizes almost automatic operation. Hereinafter, the mode of shooting operation based on a scenario may be called a scenario shooting mode. Even in scenario shooting mode, when control is executed based on each preset information, at least some of the various parameters such as the main camera position, pan/tilt, zoom, and focus are controlled while moving between locations. good. Operations related to operation control and scenario editing using the scenario DB 48c will be further described later.

Refer to FIG. 10 again. The real-time operation section 46c controls the camera units 10a and 10b via the unit control section 41 in accordance with the operator's real-time location and position operations (hereinafter referred to as real-time operations). The real-time operation may be performed by interruption during execution of operation control based on the preset DB 48b and during execution of operation control based on the scenario DB 48c. Further, the real-time operation unit 46c may store the details of operations performed by the operator during real-time operations in the storage unit 48 as an operation log 48e. The real-time operation is also performed when registering preset information in the preset DB 48b.

The authority management section 47 manages the operation authority of the camera units 10a and 10b. Only one operator can operate one camera unit, and when a plurality of operators attempt to operate one camera unit, exclusive control is performed by the authority management section 47 . For example, when the operator system 30a has the operation authority for the camera unit 10a and the operator system 30b attempts to operate the camera unit 10a, the authority management unit 47 permits or denies the operation by the operator system 30b.

In the above case, the authority management unit 47 requests the operator system 30b to operate the camera unit 10a from the operator system 30a in order to smoothly transfer the operation authority between the operator systems 30a and 30b. You may notify us that you are doing so. Further, when the operator system 30a notifies that it approves the transfer of the operation authority, the authority management unit 47 releases the operation authority of the operator system 30a and sets the operation authority of the camera unit 10a to the operator system 30b. good too. In this case, the authority management section 47 suspends the operation of the camera unit 10a in order to avoid malfunction.

The above operation authority is authority for operating the camera unit 10a. In addition, the operator may be set with authority for registration and deletion in the preset DB 48b, authority for registration and deletion in the scenario DB 48c, and the like. Operation authority and other authority are managed by operation authority information 48d including contents as shown in FIG. FIG. 25 is a chart for explaining operation authority information according to this embodiment.

In the example of FIG. 25, the authority STD is set for the operator OP1, and the authority EDIT is set for the operator OP2. STD is a privilege that allows running presets, running scenarios, and running real-time operations. EDIT is an authority that permits the execution of presets, the execution of scenarios, and the execution of real-time operations, as well as the editing of the preset DB 48b and the scenario DB 48c.

Further, in the example of FIG. 25, the operator system 30a used by the operator OP1 has the operation authority for UN1, and the operator system 30b used by the operator OP2 has the operation authority for UN2. In this case, due to exclusive control by the authority management unit 47, the operator system 30a is not set with the UN2 operation authority, and the operator system 30b is not set with the UN1 operation authority. In this way, by performing exclusive control based on operation authority, malfunction of the robot camera and confusion at the shooting site can be avoided, and operational safety can be ensured.

[4. operator system]
Next, functions of the operator system according to this embodiment will be described with reference to FIG. FIG. 26 is a block diagram for explaining the functions of the operator system according to this embodiment. Since the main configurations of the operator systems 30a and 30b are the same, the operator system 30a will be described in detail, and the description of the operator system 30b will be omitted.

As shown in FIG. 26, the operator system 30a has an operation unit 31, a display unit 32, a control unit 33, a sensor unit 34, a storage unit 35, an audio input unit 36, and an audio output unit 37. . The element P1 of the operator system 30a can be implemented using an information processing device such as a PC (Personal Computer), tablet terminal, smart phone, HMD, smart glasses, or the like. Also, the element P2 of the operator system 30a can be implemented using the information processing device of the element P1 or an audio processing device such as an intercom.

(Controller implementation form)
For example, the operation unit 31 may be a joypad type controller 31b as shown in FIG. The controller 31b is an example of operation means that provides a UI (User Interface) of the operator system 30a. FIG. 27 is a diagram schematically showing an example of a controller that can be used as the UI of the operator system according to this embodiment.

As shown in FIG. 27, the controller 31b has operation means 331, . The operating means 331, 335, 336, 337, and 338 are pushable button-type operating means. The operating means 332 and 334 are lever-type operating means capable of pointing in any direction. The operation means 333 is a cross key type operation means capable of independently instructing up, down, left and right. It should be noted that the controller 31b may be further provided with other operating means on the rear surface or the like. Independent functions can be assigned to the operation means 331, . . . , 338 and other operation means of the controller 31b.

Functions that can be assigned to the operation means include, for example, a function to move the camera unit in the studio coordinate system (location operation function), a function to move the TCP in the unit coordinate system (position operation function), and a function to move the camera unit in the optical axis direction in the camera coordinate system. There is a function to move the (pan/tilt operation function). In addition, the function to operate camera parameters such as zoom, focus, iris, and gain of the main camera (camera operation function), the function to register the current location (location registration function), the function to register the current position (position registration function).

The above position operation function may include a function of switching to operation by the operation interface 301 shown in FIGS. 16(b), 17(b), and 18(b). In this case, the position manipulation function includes position manipulation within the uZ-uX plane (first manipulation: FIG. 16(a)), position manipulation within the uY-uZ plane (second manipulation: FIG. 17(a) ) and the position operation (third operation: FIG. 18(a)) in the uX-uY plane.

In addition, the above operation means include a function of switching to an operation based on movement control based on a fixed point (eTCP) shown in FIG. (rotation angle, etc.), and a function of executing position and pan/tilt control as shown in FIG. 19 based on eTCP.

In addition, the above operation means include a function to select a camera unit (camera selection function), a function to switch between real-time operation mode and preset shooting mode (mode switching function), and selection of preset information to be used in preset shooting mode. A function (preset selection function) may be assigned. The mode switching function may switch between the scenario shooting mode and another operation mode. Other modes of operation include a real-time operation mode or preset capture mode, an emergency stop mode, and the like.

In addition, the function of switching the video displayed on the display unit 32 (video switching function), the function of switching the form of the UI displayed on the display unit 32 (UI switching function), the audio input unit 36 and the audio output unit 37 A function (call function) for talking with an operator or the like may be assigned.

The image switching function is a function for switching the image displayed in the main display area of the display unit 32 from the main image output from the currently operated camera unit to the main image output from another camera unit. good. Further, the image switching function may be a function of switching the image displayed in the main display area of the display unit 32 between the main image, the eye-view image, and the rear-view image. Further, the video switching function may be a function of switching between a plurality of rear-view videos output from different rear-view cameras.

The above UI switching function may be a function of switching between a plurality of UIs with different combinations of images displayed on the display unit 32, combinations of information, and different arrangements of images and information. The above-mentioned call function may be a function for calling an operator selected from a list of other operators who have operation authority for other camera units, or a function for all operators included in the list. It may also be a function that allows voice to be transmitted to the device. Further, the call function may include a function of selecting whether or not to permit a call, a function of recording a call log, and a function of notifying another operator whether or not a call is possible.

It is possible to arbitrarily decide which function is assigned to which operating means. Further, it is not necessary to allocate all the above-mentioned functions illustrated, and functions other than the above-mentioned functions may be allocated. Also, any function may be assigned to an operation that combines a plurality of operation means. For example, a method of assigning a position operation function to an operation of operating the operating means 332 alone and assigning a location operating function to an operation of operating the operating means 332 while pressing the operating means 336 can be applied.

As described above, since it is possible to arbitrarily decide which function is assigned to which operation means, the form of the operation unit 31 is not limited to a joypad type like the controller 31b. For example, the form of the operation unit 31 may be a type like the controller 31c shown in FIG. FIG. 28 is a diagram schematically showing another example of a controller that can be used as the UI of the operator system according to this embodiment.

The controller 31 c illustrated in FIG. 28 has operation means 341 , 342 , 343 , 344 and 345 . The operation means 341 and 344 are dial-type operation means with a limited rotatable range. The operation means 345 is a dial-type operation means whose rotatable range is not restricted or whose rotatable range is wider than that of the operation means 341 and 344 . The operation means 342 is a lever-type operation means that can move up and down. The operation means 343 is a keyboard type operation means having a plurality of buttons.

Some interfaces used as switcher consoles for switching images have a form similar to that of the controller 31c. For example, the various functions described above for operating the camera unit may be assigned to some operating means of such an operator console. For example, an assignment method is applied in which a pan angle changing operation is assigned to an operation of operating the operating means 341 while a specific button included in the operating means 343 is pressed, and a tilt angle changing operation is assigned to an operation of operating the operating means 342. can do.

Moreover, as a form of the operation unit 31, a form using a gesture operation type device as shown in FIGS. 29(a) and (b) is also applicable. FIGS. 29A and 29B are diagrams schematically showing an example of a gesture operation type device that can be used as the UI of the operator system according to this embodiment.

In this mode, as shown in FIG. 29A, the operator OP wears smart glasses 351 that are one form of the display unit 32, wears an intercom 352 that functions as the element P2 of the operator system 30a, and operates the operation unit. As 31, controllers 353 and 354 are used. For example, the operator OP holds and operates the controller 353 with his right hand, and holds and operates the controller 354 with his left hand. Note that the controllers 353 and 354 may be attached to the operator OP's body (arms, etc.) using fixing means such as bands.

Controllers 353 and 354 are equipped with motion sensors such as acceleration sensors. Further, as shown in FIG. 29B, the controller 353 is provided with a button RC, and the controller 354 is provided with a button CC. Various functions such as the location operation function, the position operation function, and the pan/tilt operation function described above can be assigned to movements of the controllers 353 and 354, operations of the buttons RC and CC, and combinations of the movements and operations. Also, functions may be assigned to specific gestures by the controllers 353 , 354 .

For example, when the operator OP moves the controller 353 in the Fw direction of FIG. 29(b), the location moves in the +sY direction, when moved in the Bw direction, the location moves in the −sY direction, and when moved in the R direction, the location moves in the +sX direction. , and a function can be assigned such that moving in the L direction moves the location in the -sX direction. Alternatively, a function may be assigned such that when the button RC is pressed once, movement of the camera unit is locked at the current location, and location movement operations are disabled until the button RC is pressed again.

Also, when the operator OP moves the controller 354 in the Fw direction of FIG. 29(b), the position moves in the +uY direction, when moved in the Bw direction, the position moves in the -uY direction, and when moved in the R direction, the position moves in the +uX direction. , and moving in the L direction moves the position in the -uX direction, and moving in the direction perpendicular to the Fw/Bw and R/L directions (perpendicular to the paper surface) moves the position along the uZ axis. You can assign functions like Also, a function may be assigned such that when the button CC is pressed once, the current position is fixed and the optical axis direction in the camera coordinate system moves following the movement of the controller 354 .

Arbitrary functions can be assigned to the various operations described above, without being limited to the example of assignment described above. For example, the location movement in the studio coordinate system and the position movement in the unit coordinate system described above are replaced with the movement in the shooting direction in the camera coordinate system. May be assigned to movement. It is also possible to assign the three-plane independent control mechanism described above to the movements of the controllers 353 and 354 . Furthermore, gestures that draw circles, gestures that draw specific letters and symbols (alphabet, Greek letters, mathematical symbols, etc.), gestures that draw patterns, and combinations of these gestures and button RC and CC operations are optionally selected. function may be assigned. These modifications also naturally belong to the technical scope of this embodiment.

Further, as the form of the operation unit 31, a form using a seated operation unit as shown in FIG. 30 is also applicable. FIG. 30 is a diagram schematically showing an example of a seated operation unit that can be used as the UI of the operator system according to this embodiment.

In this form, the operator OP sits in front of displays 361a, 361b, and 361c, which are forms of the display unit 32, and wears an intercom 363 that functions as an element P2 of the operator system 30a. Operating means 364, 365, 366 and 367 are operated. The operating means 366 is a foot controller. The operation means 365 is a lever type operation means. The operation means 364 is a button type operation means. The operation means 367 is a dial type operation means.

As with the joypad type controller 31b described above, it is possible to assign any function to each operating means. The displays 361a, 361b, and 361c may display a main image output from the currently operated camera unit, a main image output from another camera unit, an eye-view image, a rear-view image, and the like. These images may be displayed on touch panel 362 . Therefore, a function of switching which image is displayed on which device among the displays 361a, 361b, 361c and the touch panel 362 may be assigned to the operation means.

By the way, a computer such as a PC or a tablet terminal can be used as the operator system 30a. When using a PC, the operation unit 31 is a keyboard or mouse, and the display unit 32 is a display such as LCD or ELD. In this case, the various functions described above can be assigned to keyboard and mouse operations. Also, the various functions described above may be assigned to the operation of the GUI object displayed on the display. When using a tablet terminal, the operation unit 31 and the display unit 32 are touch panels, and the various functions described above are assigned to operations of GUI objects displayed on the touch panels.

(GUI)
FIG. 31 exemplifies one aspect of the GUI displayed on the display unit 32 as the operation system of the camera unit described above. FIG. 31 is a schematic diagram for explaining the UI (real-time operation mode) of the operator system according to this embodiment.

31, the display unit 32 includes tally notification areas 311a, . . . , 311d, image display areas 312a, . , . . . , 314d are displayed. The display unit 32 also displays a system notification area 315 , a mode switching button 317 , a message area 318 , and a mode display area 319 .

, BTN8, UP1, UP2, DN1, DN2, cursor objects CUR1, . . . , CUR4, circle button objects CCL1, . is displayed. These objects are examples of GUI objects that constitute the operation system of the camera unit. As with the various controllers described above, these GUI objects can be assigned various functions.

The tally notification areas 311a, . . . , 311d correspond to the video display areas 312a, . In the example of FIG. 31, since the main image displayed in the image display area 312b is on tally, the tally notification area 311b has a different display form (hatched in the drawing) from the tally notification areas 311a, 311c, and 311d. there is When there is an image display area in which a rear-view image or an eye-view image is displayed, the display form of the tally notification area corresponding to the image display area may be the off-tally display form.

A main image, a rear-view image, or an eye-view image is displayed in the image display areas 312a, . . . , 312d. The camera identification display areas 313a, . . . , 313d and the operator identification display areas 314a, . In the camera identification display areas 313a, . The operator identification display areas 314a, . When the image in the corresponding image display area is the rear view image, the operator's identification information is not displayed.

Notifications from the control system 40 are displayed in the system notification area 315 . For example, the system notification area 315 displays information such as presence/absence of messages, various notifications regarding transfer of operation authority, and error notifications. The video display area 316 displays the main video output from the currently operated camera unit. Further, when an operation (touch, click, etc.) is performed on the image display areas 312a, . good.

A mode switching button 317 is a button object for switching to sub-view mode. When the mode switching button 317 is pressed, the display form is switched to the sub-view mode as shown in FIG. FIG. 32 is a schematic diagram for explaining the UI (sub-view mode) of the operator system according to this embodiment. As shown in FIG. 32, in the sub-view mode, a first image display area 321 and a second image display area 322 are displayed in a large size on the display unit 32 in place of the operating GUI objects such as the button object BTN1.

An eye-view image or a rear-view image (sub-view image) is displayed in the first image display area 321 . For example, the image displayed in the image display area 312 d is displayed in the first image display area 321 . A main image is displayed in the second image display area 322 . For example, the second image display area 322 displays the on-tally main image or the main image output from the currently operated camera unit. Note that the main image may be displayed in the first image display area 321 and the sub-view image may be displayed in the second image display area 322 . When the mode switching button 323 is pressed, the normal display of FIG. 31 is restored.

A message area 318 displays messages sent by other operators. The message area 318 may also allow the operator to enter a message. For example, when a communication tool is activated by the communication tool providing unit 45 of the control system 40 and a chat is possible with another operator, the operator can enter characters in the message area 318 to communicate with the other operator. You can chat with the operator. Mode display area 319 displays the current operating mode.

, BTN8, UP1, UP2, DN1, DN2, cursor objects CUR1, . . . , CUR4, circle button objects CCL1, . An example is given.

In this example, the button object BTN1 is assigned the function of switching to the real-time operation mode in which the operator manually operates the camera unit. The button object BTN2 is assigned the function of moving the TCP to the home position of the preset unit coordinate system.

Cursor object CUR1 is assigned the function of moving TCP in the +uZ direction. Cursor object CUR2 is assigned the function of moving TCP in the -uZ direction. The cursor object CUR3 is assigned a function of rotating TCP counterclockwise on the uX-uY plane with the uZ axis as the rotation axis (left rotation function). The cursor object CUR4 is assigned a function of rotating the TCP clockwise on the uX-uY plane with the uZ axis as the rotation axis (right rotation function).

The circle button object CCL1 is assigned the function of moving TCP in the +uY direction. Circle button object CCL2 is assigned the function of moving TCP in the -uY direction. Circle button object CCL3 is assigned the function of moving TCP in the +uX direction. Circle button object CCL4 is assigned the function of moving TCP in the -uX direction. Also, the function of temporarily locking the current position may be assigned to the center button object CC1.

The circle button objects CCL1, . These functions may be switched by, for example, pressing the button object BTN3. The function of the donut-shaped object assumes an orthogonal coordinate system in which the center of the donut-shaped object is the origin and the direction of the circle button object CCL1 corresponds to the uY direction. to move.

, CUR4 and circle button objects CCL1, . Further, when the button object BTN4 is pressed, the movement of the TCP in response to the operation of the circle button objects CCL1 and CCL2 may be switched in the +uZ direction and the -uZ direction, respectively.

Button object BTN5 is assigned the function of registering the current position. Button object BTN6 is assigned a function of automatically adjusting the orientation of the main camera horizontally. A zoom function is assigned to the button objects UP1 and DN1. For example, while the button object UP1 is pressed, the lens moves toward the telephoto end, and while the button object DN1 is pressed, the lens moves toward the wide end. A focus function is assigned to the button objects UP2 and DN2. For example, while the button object UP2 is pressed, the lens is moved so that the focus position approaches the camera, and while the button object DN2 is pressed, the lens is moved so that the focus position moves away from the camera.

A tilt control function is assigned to the circle button objects CCL5 and CCL6. For example, pressing the circle button object CCL5 increases the tilt angle θx, and pressing the circle button object CCL6 decreases the tilt angle θx. A pan control function is assigned to the circle button objects CCL7 and CCL8. For example, pressing the circle button object CCL7 increases the panning angle θz, and pressing the circle button object CCL8 decreases the panning angle θz. Also, the center button object CC2 may be assigned a function of temporarily locking the current pan/tilt state.

It should be noted that the circle button objects CCL5, . For example, pressing the button object BTN7 may switch between a mode functioning as an independent button object and a mode functioning as a doughnut-shaped object. Further, the roll control function may be assigned to the circle button objects CCL7 and CCL8 when the button object BTN8 is pressed. Also, when a long press operation is performed on these button objects, the corresponding movements may be performed continuously.

By using the above GUI, the camera unit can be suitably operated.

Here, a GUI used for operations in the preset shooting mode and the scenario shooting mode will be described with reference to FIGS. 33 and 34. FIG. FIG. 33 is a schematic diagram for explaining the UI (preset imaging mode) of the operator system according to this embodiment. FIG. 34 is a schematic diagram for explaining the UI (scenario shooting mode) of the operator system according to this embodiment.

In the preset shooting mode, a preset display area 371 is displayed on the display section 32 as shown in FIG. The preset display area 371 displays thumbnails corresponding to preset information and parameter sets included in the preset information. The preset information displayed in preset display area 371 is obtained from control system 40 . In the example of FIG. 33, thumbnails and parameter sets of PS001, . . . , PS007 are displayed.

The operator selects the preset information displayed in the preset display area 371 with the operating body 302, thereby executing the camera unit based on the preset information. For example, when the operator selects PS006, the control unit 33 of the operator system 30a notifies the control system 40 of information on PS006. The control system 40 that has received the information of PS006 controls the camera units for which the operator system 30a is authorized to operate based on the parameter set of PS006 by the function of the preset execution section 46a.

In scenario shooting mode, a scenario display area as shown in FIG. The scenario display area includes a title display area 381, a timeline display area 382, unit control areas 383a, 383b, 383c, and situation display areas 384a, 384b, 384c, 384d.

A title display area 381 displays a scenario title for identifying a scenario. A timeline is displayed in the timeline display area 382 . Preset information used for controlling UN1 is displayed in the unit control area 383a. Preset information used for controlling UN2 is displayed in the unit control area 383b. Preset information used for controlling UN3 is displayed in the unit control area 383c.

The preset information displayed in the unit control areas 383a, 383b, and 383c is displayed in association with the time on the timeline corresponding to the execution timing. The status display areas 384a, 384b, 384c, and 384d display the progress status of the operation at the corresponding time. For example, "DONE" is displayed when the operation is completed, and "READY" is displayed when the operation is in progress or is being prepared. In the example of FIG. 34, the portion currently executing the action is displayed with a bold frame.

When the scenario is executed, actions based on the preset information are executed in order along the timeline. In the example of FIG. 34, UN1 starts an operation based on PR0018 at the set time within the time frame of 00:00:00, and UN3 starts an operation based on PR011 at the set time within the same frame. At the set time within the time frame of 00:10:00, UN1 starts operating based on PR021, and at the set time within the same frame, UN2 starts operating based on PR015. In this manner, actions are automatically advanced based on the scenario information displayed side by side in the scenario display area. Note that the execution start time of each preset operation may be different or the same. Also, the execution time of each preset operation may be different or the same. These settings can be defined using preset information. The video output may be performed continuously at UN1, UN2, . . . and the video output may be controlled by a switcher.

By using the GUI described above, the operator can perform operations in the preset shooting mode and the scenario shooting mode.

(line-of-sight detection)
Refer to FIG. 26 again. The control unit 33 transmits a control signal for controlling the camera unit to the control system 40 in accordance with an input to the operation unit 31 and controls the camera unit in cooperation with the control system 40 . Further, the control unit 33 can execute the display control of the GUI described above. Further, the control unit 33 may store the contents of operations performed by the operator in the storage unit 35 as an operation log, and may provide the operation log to the control system 40 .

The sensor unit 34 includes a motion sensor such as an acceleration sensor, a magnetic sensor, and a gyroscope, and detects the line-of-sight direction of the operator based on the output of the motion sensor. The line-of-sight direction of the operator may be set in the direction from the operator's face to the screen of the display unit 32 . By setting the line-of-sight direction when the operator is in an upright posture as the reference (front), the current line-of-sight direction can be obtained as the amount of deviation from the front based on the output of the motion sensor.

The sensor unit 34 may be a sensor device worn on the operator's head, or may be a motion sensor built into an HMD or smart glasses. As a modification, the sensor unit 34 may be a camera that captures an image of the operator's head instead of the motion sensor. In this case, the line-of-sight direction can be detected by image processing from an image of the operator's head. This image processing may be performed by the control unit 33 . The line-of-sight direction information output from the sensor unit 34 is transmitted to the control system 40 via the control unit 33 .

The control system 40 that has received the line-of-sight direction information transmits an eye-view image corresponding to the operator's field of view cut out from the output image of the eye-view camera 12 to the operator system 30a. The control unit 33 causes the display unit 32 to display the eye-view image received from the control system 40 . An omnidirectional image may be obtained from the control system 40 , and the eye-view image corresponding to the line-of-sight direction may be cut out by the control unit 33 and displayed on the display unit 32 .

The functions of the operator system 30a described above may be realized by the control unit 33 executing the program 35a stored in the storage unit 35. FIG. The program 35a may be stored in advance in the storage unit 35, or may be acquired from the control system 40 or another server device (not shown) on the network.

[5. Processing flow]
Next, the flow of processing in the remote camera system according to this embodiment will be described. Hereinafter, a processing flow regarding preset registration, a processing flow regarding trace registration, a processing flow regarding scenario setting, and a processing flow regarding setting and transfer of operation authority will be described in order.

(Regarding preset registration)
First, the flow of processing relating to preset registration according to the present embodiment will be described with reference to FIG. FIG. 35 is a flowchart for explaining the flow of processing relating to preset registration according to this embodiment. A case where the operator OP performs preset registration using the operator system 30a will be described below.

(S101) The operator OP requests the control system 40 to start preset registration via the operator system 30a. Upon receiving this request, the control system 40 confirms that the authority (EDIT) permitting preset registration is set for the operator OP. Here, it is assumed that the operator OP has the EDIT authority.

(S102-S104) The operator OP inputs the location, position and camera parameters to be registered to the operator system 30a. The location can be specified by the coordinate values of the studio coordinate system in which the camera units are arranged. The position can be designated by the coordinate values of the unit coordinate system in which the TCP is arranged. Camera parameters include tilt angle θx, pan angle θz, zoom value, and the like. For camera parameters that are not input by the operator OP, preset default values may be used. Operator system 30a sends control signals to control system 40 indicating the location, position, and camera parameters entered.

(S105) The control system 40 moves the camera unit to the location indicated by the control signal received from the operator system 30a. Control system 40 also moves the TCP to the position indicated by the received control signal. In addition, the control system 40 sets the pan/tilt direction and sets the set values such as zoom to the main camera based on the camera parameters indicated by the received control signal. Then, the control system 40 starts the operation of the camera unit after setting, and transmits the main image output from the camera unit to the operator system 30a.

(S106, S107) The operator OP adjusts the position and camera parameters as necessary while viewing the main image transmitted from the control system 40. FIG. Once the position and camera parameters are determined, the operator OP performs a registration operation. In response to this registration operation, the operator system 30a sends a control signal instructing registration to the control system 40. FIG.

(S108) The control system 40 creates a thumbnail from the main video currently output from the camera unit in accordance with the control signal instructing registration. For example, the control system 40 cuts out still images from the main video, adjusts the resolution of the still images, and generates thumbnails. Note that the control system 40 may compress/encode the still image based on a predetermined compression/encoding method.

(S109) The control system 40 registers the location, position, camera parameters, and thumbnail generated in S108 at the time of registration operation by the operator OP in the preset DB 48b as preset information.

(S110) When the registration of the preset information ends (for example, when the operator OP performs a termination operation), the series of processes shown in FIG. 35 ends. If the registration of preset information is to be continued, the process proceeds to S102.

(Regarding trace registration)
Next, the flow of processing relating to trace registration according to this embodiment will be described with reference to FIG. FIG. 36 is a flowchart for explaining the flow of processing relating to trace registration according to this embodiment. A case where the operator OP performs trace registration using the operator system 30a will be described below.

The preset information described so far indicates one state of the camera unit. Here, trace registration for pre-registering the state transition of the camera unit from one state (state A) to another state (state B) will be described. Trace registration is also a kind of preset registration in the sense of pre-registering information about the state of the camera unit.

(S121) The operator OP requests the control system 40 to start trace registration via the operator system 30a. Upon receiving this request, the control system 40 confirms that the authority (EDIT) permitting trace registration is set for the operator OP. Here, it is assumed that trace registration is also permitted with EDIT authority, and that EDIT authority is set for the operator OP.

(S122-S124) The operator OP sets the control mode of the camera unit to the real-time operation mode, moves the camera unit to a desired location, and adjusts the position and camera parameters. When this work is completed, the operator OP executes the start operation of trace registration. Then, the operator OP starts camera work by real-time operation.

(S125) The control system 40 performs chronological recording of the location, position, and camera parameters after the operator OP performs the trace registration start operation. That is, control system 40 records the behavior of the camera unit. Thereby, the operator OP's camerawork is saved in the control system 40 as a trace.

(S126) The operator OP may perform preset registration during the trace registration operation. When the operator OP performs the preset registration operation, the process proceeds to S127. On the other hand, when the operator OP has not performed the preset registration operation, the process proceeds to S129.

(S127, S128) The control system 40 creates thumbnails from the main video at the time of preset registration operation. In addition, the control system 40 registers the location, position, pan/tilt, and other camera parameters of the camera unit at the time of preset registration operation, and the generated thumbnail in the preset DB 48b.

(S129) When the operator OP performs an operation to end the trace registration, the control system 40 uses time-series information indicating the state of the camera unit recorded between the start operation and the end operation of the trace registration as trace information. It is held in the storage unit 48 . On the other hand, if the operator OP has not performed an operation to end the trace registration, the process proceeds to S125.

The above trace information, like preset information, can be incorporated into scenarios. For example, by preparing a movement route for safely moving the camera unit as trace information and incorporating it into the scenario, safer scenario shooting becomes possible. Also, by using trace information, it is possible to move a specific camera unit to the desired location in advance, even during preset shooting. In this way, the preset information can be handled in the same manner as the preset information described so far, and can be used for preset shooting and scenario shooting.

(About scenario settings)
Next, with reference to FIG. 37, the flow of processing relating to scenario setting according to this embodiment will be described. FIG. 37 is a flowchart for explaining the flow of processing relating to scenario setting according to this embodiment. A case where the operator OP uses the operator system 30a to set a scenario will be described below.

(S131) The operator OP requests the start of scenario setting for the control system 40 via the operator system 30a. Upon receiving this request, the control system 40 confirms that the authority (EDIT) permitting scenario setting is set for the operator OP. Here, it is assumed that scenario setting is permitted with EDIT authority, and that EDIT authority is set for the operator OP.

(S132) The operator OP can set a new scenario or edit an existing scenario. When setting a new scenario, the process proceeds to S133. On the other hand, when editing an existing scenario, the process proceeds to S135.

(S133) The operator system 30a receives notification from the control system 40 that the operator OP has authority, and displays the timeline and preset list corresponding to each camera unit based on the preset information acquired from the control system 40. . For example, the operator system 30a displays a timeline display area 382 and unit control areas 383a, 383b, and 383c as shown in FIG. 34, and a preset information list (preset list) such as the preset display area 371 in FIG. may be displayed.

(S134) The operator OP selects preset information from the preset list and arranges the selected preset information on the timeline corresponding to each camera unit. In this way, a scenario can be generated by repeating the selection and arrangement of preset information. When the process of S134 is completed, the process proceeds to S137.

(S135, S136) The operator system 30a receives a notification from the control system 40 that the operator OP has authority, and based on the scenario information of the existing scenario acquired from the control system 40, the preset information is arranged in each camera unit. View the corresponding timeline. For example, the operator system 30a displays a timeline display area 382 and unit control areas 383a, 383b, 383c as shown in FIG. The operator OP changes the arrangement of the displayed preset information.

(S137) The control system 40 acquires the scenario information set by the operator system 30a, and based on the scenario information, determines whether or not the flow lines of the camera units interfere with each other when the camera units are moved in the studio. . For example, the control system 40 executes a movement simulation of each camera unit based on the scenario information and checks whether the camera units will collide with each other. If the flow lines of the camera units do not interfere, the process proceeds to S138. On the other hand, if the flow lines of the camera units interfere, the process advances to S135 to request the operator OP to edit the current scenario.

In the example of FIG. 37, the determination condition in S137 is the presence or absence of interference between flow lines, but this may be replaced with another determination condition, or another determination condition may be added to the presence or absence of interference between flow lines. Other conditions include, for example, that the main image does not include any unnecessary objects such as other camera units (that is, that there is no cut-off). Of course, other determination conditions may be set in order to ensure a safe shooting environment, obtain stable images, and the like. Such modifications naturally belong to the technical scope of the present embodiment.

(S138) The control system 40 saves the set scenario information. When the process of S138 is completed, the series of processes shown in FIG. 37 ends.

(Regarding setting and transfer of operation authority)
Next, with reference to FIG. 38, the flow of processing regarding setting and transfer of operation authority according to this embodiment will be described. FIG. 38 is a sequence diagram for explaining the flow of processing related to setting and transferring operation authority according to this embodiment. A case where the operator OP1 operates the operator system 30a and the operator OP2 operates the operator system 30b will be described below.

(S141) The operator OP1 operates the operator system 30a to request the control system 40 to operate UN1.

(S142) The control system 40 refers to the operation authority information 48d and confirms whether UN1 currently has the operation authority. Here, it is assumed that the operation authority is not set for UN1. In this case, the control system 40 determines that the operation authority of UN1 may be set to the operator system 30a.

(S143) The control system 40 sets UN1 operation authority for the operator system 30a, and updates the operation authority information 48d.

(S144) The control system 40 transmits to the operator system 30a a notification that the operation authority for UN1 has been set.

(S145) When the operator system 30a receives the notification from the control system 40 that the operation authority for UN1 has been set, the operator system 30a displays and notifies the operator OP1. An operator OP1 operates UN1 using an operator system 30a.

(S146) If the operator OP2 desires to operate UN1 while the operator OP1 is operating UN1, the operator OP2 operates the operator system 30b to request the control system 40 to operate UN1.

(S147) The control system 40 refers to the operation authority information 48d and confirms whether or not UN1 currently has the operation authority. Here, the operation authority of UN1 is set for the operator system 30a. In this case, the control system 40 determines that the operation authority for UN1 is set to the operator system 30a.

(S148) The control system 40 transmits to the operator system 30b a notification that the operator system 30a has been granted the UN1 operation authority (that the operator OP1 is in use).

(S149) The control system 40 transmits to the operator system 30a a notification that the operator system 30b (operator OP2) has requested the operation authority of UN1.

(S150) Operators OP1 and OP2 negotiate to transfer the operation authority of UN1 from operator OP1 to operator OP2. For example, the control system 40 may provide a communication tool for chatting between the operator systems 30a, 30b so that the operators OP1, OP2 can negotiate. Also, the control system 40 may store the content of the chat in the storage unit 48 as a communication log 48f.

(S151) When the operator OP1 agrees to transfer the operation authority of UN1, the operator system 30a requests the control system 40 to release the operation authority of UN1. Note that the operator system 30a may send a notification to the control system 40 to the effect that it approves the release of the operation authority for the UN1.

(S152) The control system 40 that has received the operation authority release request for the UN1 from the operator system 30a suspends the operation of the UN1. As a result, the operation on UN1 is disabled while the operation authority is being transferred.

(S153, S154) The control system 40 releases the operation authority for UN1 set in the operator system 30a, and updates the operation authority information 48d. Then, the control system 40 sends a notification to the operator system 30a that the release of the operation authority for UN1 has been completed.

(S155, S156) The control system 40 sets the operation authority for UN1 to the operator system 30b, and updates the operation authority information 48d. The control system 40 then sends a notification to the operator system 30b to the effect that the setting of the operation authority for UN1 has been completed.

(S157) When the operator system 30b receives the notification from the control system 40 that the operation authority for UN1 has been set, the operator system 30b displays and notifies the operator OP2. An operator OP2 operates UN2 using an operator system 30b.

As described above, the control system 40 manages the operation authority and implements exclusive control for assigning one operator to one camera unit, thereby avoiding confusion at the shooting site and controlling the remote control of the camera unit. It becomes possible to perform safe shooting by operation. In addition, by allowing operators to appropriately negotiate with each other when transferring the operation authority, the operator can be replaced smoothly.

[6. Application to 3D simulation]
So far, the explanation has been given on the assumption that the camera unit is actually installed in the studio, but it is also possible to create the remote camera system 5 described above in the virtual space by 3D simulation. In this case, the camera units 10a and 10b and the rear view cameras 20a and 20b described above are constructed as virtual objects in the virtual space, and the virtual objects are operated by the operator systems 30a and 30b described above. In this scheme, control of the virtual object is performed by control system 40 . Thus, the remote camera system 5 can also be virtualized by 3D simulation. Such application examples naturally belong to the technical scope of the present embodiment.

[7. hardware]
Next, hardware capable of realizing the functions of the control system according to this embodiment will be described with reference to FIG. FIG. 39 is a block diagram for explaining hardware capable of realizing the functions of the control system according to this embodiment.

The functions of the control system 40 can be implemented using, for example, hardware resources shown in FIG. That is, the functions of the control system 40 are implemented by controlling the hardware shown in FIG. 39 using a computer program.

As shown in FIG. 39, this hardware mainly has a processor 40a, a memory 40b, a display I/F (Interface) 40c, a communication I/F 40d and a connection I/F 40e.

The processor 40a may be a CPU (Central Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), GPU (Graphic Processing Unit), or the like. The memory 40b may be ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, or the like.

The display I/F 40c is an interface for connecting display devices such as LCD (Liquid Crystal Display) and ELD (Electro-Luminescence Display). The communication I/F 40d is an interface for connecting to a wired and/or wireless network. The communication I/F 40d may be connected to a wired LAN (Local Area Network), a wireless LAN, an optical communication network, or the like.

The connection I/F 40e is an interface for connecting external devices. The connection I/F 40e may be a USB (Universal Serial Bus) port, IEEE1394 port, SCSI (Small Computer System Interface), or the like. An input interface such as a keyboard, mouse, touch panel, or touch pad may be connected to the connection I/F 40e. A portable recording medium 40f such as a magnetic recording medium, an optical disk, a magneto-optical disk, or a semiconductor memory may be connected to the connection I/F 40e.

The processor 40a may read the program stored in the recording medium 40f, store it in the memory 40b, and control the operation of the control system 40 according to the program read from the memory 40b. A program that controls the operation of the control system 40 may be stored in advance in the memory 40b, or may be downloaded from the network via the communication I/F 40d.

The functions of the unit control section 41, the main video output section 42, the eye view providing section 43, the rear view providing section 44, the communication tool providing section 45, the camera work execution section 46, and the authority management section 47, which have already been described, are mainly: It can be implemented using the processor 40a described above. The functions of the storage unit 48 already described can be realized mainly using the memory 40b described above.

Here, the configuration example of FIG. 39 is shown as a hardware configuration capable of realizing the functions of the control system 40. For example, at least part of the functions of the control boxes 14 of the camera units 10a and 10b, the operator system 30a, The functions of 30b can also be implemented using the hardware configuration illustrated in FIG. For example, the functions of the control box 14 can be implemented by the processor 40a described above.

Also, the functions of the operation unit 31 included in the operator systems 30a and 30b can be realized by various input interfaces connected to the connection I/F 40e described above. Moreover, the function of the display unit 32 can be realized by various display devices connected to the display I/F 40c described above. Also, the functions of the control unit 33 can be realized by the above-described processor 40a. Also, the function of the storage unit 35 can be realized by the above-described memory 40b. Also, the functions of the sensor unit 34, the audio input unit 36, and the audio output unit 37 can be implemented by various sensor devices (not shown) and audio devices that can be connected to the connection I/F 40e described above.

As described above, the embodiments of the present disclosure have been described with reference to the accompanying drawings, but the present disclosure is not limited to such examples. It is obvious that a person skilled in the art can conceive various modifications or applications within the scope described in the claims, and these naturally belong to the technical scope of the present disclosure.

Additional remarks about technical matters according to the embodiments of the present disclosure will be disclosed below. The contents of the supplementary notes disclosed here simply indicate the technical idea of the present disclosure, and are not intended to limit the technical matters of the present disclosure. Since the disclosure is made to assist those skilled in the art in devising various modifications and applications from the matter, modifications and Application examples also naturally belong to the technical scope of the present disclosure.

[A. About control rights]
(Appendix A1) Including a remotely operable camera unit, a control system for controlling the camera unit, and first and second operator systems for operating the camera unit via the control system ,
When the control system receives a request for setting the operation authority from the second operator system in a state where the operation authority for the camera unit is set for the first operator system, the control system After receiving a response from the operator system to the effect that the transfer of the operation authority is permitted, the operation authority set in the first operator system is released, and the operation authority is given to the second operator system. Configure a remote camera system.

According to this configuration, when the operator who operates the camera unit is replaced, the operation authority is transferred in a state where the operator who is currently operating is surely aware of the replacement, so that confusion accompanying the replacement of the operator can be avoided. can be done. For example, even if the intercom is used to notify a change of operator, the operator currently operating may fail to hear the notification. In that case, the operator who is currently operating will not be aware of the replacement, and if the operating authority is transferred to another operator in that state, the operator who has been operating up to that point will suddenly lose control and cause confusion. Such unnecessary confusion can be avoided by applying the above configuration. Moreover, cooperation among staff members including operators is important at a shooting site, and ensuring mutual recognition between operators contributes to smooth progress of shooting operations.

(Appendix A2) When the control system receives the request from the second operator system, the control system transmits a notification to the effect that the request has been received from the second operator system to the first operator system, and If there is no response from the first operator system even after a predetermined time has passed since the notification, the operation authority set in the first operator system is released, and the second operator system The remote camera system according to appendix A1, wherein the operation authority is set.

(Appendix A3) When transferring the operation authority from the first operator system to the second operator system, the control system, before releasing the operation authority set in the first operator system, The remote camera according to appendix A1 or A2, wherein the operation of the camera unit is stopped and the operation of the camera unit is invalidated at the same time, and the operation of the camera unit is validated after setting the operation authority for the second operator system system.

(Appendix A4) When the control system receives the request from the second operator system, the control system transmits a notification to the effect that the request has been received from the second operator system to the first operator system, and The remote camera system of any one of Appendices A1-A3, activating a communication tool for exchanging textual information between a first operator system and said second operator system.

(Appendix A5) The remote camera system according to Appendix A4, wherein the control system stores the text information exchanged via the communication tool as a communication log.

(Appendix A6) The control system includes preset information including location information indicating the position of the camera unit at the shooting site, position information for defining the position and orientation of the main camera, and setting information of the main camera. and when the operation authority is set to the first operator system, control of the camera unit is executed based on the preset information in response to an execution request from the first operator system. The remote camera system according to any one of A5.

(Appendix A7) When the control system receives an instruction to register the preset information from the first operator system, if the registration authority of the preset information is set to the first operator system, the registration instruction generating a thumbnail of the video output from the main camera at the time of and the location information, the position information, and the setting information at the time of the registration instruction, and generating the generated thumbnail, the location information, and the position information and the setting information as the preset information.

(Appendix A8) Including a remotely operable camera unit, a control system for controlling the camera unit, and first and second operator systems for operating the camera unit via the control system The control system of a remote camera system, comprising:
When a request for setting the operation authority is received from the second operator system in a state where the operation authority for the camera unit is set in the first operator system, the operation is performed by the first operator system. After receiving a response to approve the transfer of authority, the control system releases the operation authority set in the first operator system and sets the operation authority in the second operator system. .

(Appendix A9) Including a remotely operable camera unit, a control system for controlling the camera unit, and first and second operator systems for operating the camera unit via the control system the control system of the remote camera system comprising:
When a request for setting the operation authority is received from the second operator system in a state where the operation authority for the camera unit is set in the first operator system, the operation is performed by the first operator system. releasing the operation authority set in the first operator system and setting the operation authority in the second operator system after receiving a response to approve the transfer of authority; Execute,
Permission management method.

(Appendix A10) Including a remotely operable camera unit, a control system for controlling the camera unit, and first and second operator systems for operating the camera unit via the control system to the control system of the remote camera system;
When the control system receives a request for setting the operation authority from the second operator system in a state where the operation authority for the camera unit is set in the first operator system, the first After receiving a response from the operator system to the effect that the transfer of the operation authority is permitted, the operation authority set in the first operator system is released, and the operation authority is given to the second operator system. Set the program that causes the action to take place.

[B. About the operation interface]
(Appendix B1) A camera unit having a main camera and a motion unit for controlling the position of the main camera, and an operator system used for operating the camera unit, the operator system comprising:
a first operation interface for moving the position of the main camera within a first plane including the optical axis of the main camera; and a position of the main camera within a second plane intersecting the first plane. a second operation interface for moving,
remote camera system.

According to such a configuration, since the movement of the main camera is restricted within a two-dimensional plane by operating with one operation interface, the operator can easily grasp the movement of the main camera. In addition, by providing a plurality of operation interfaces, it is possible to move within a plurality of two-dimensional planes, and a sufficient degree of freedom in positioning the main camera can be ensured. In addition, by defining the first and second planes based on the optical axis of the main camera, it becomes easier for the operator to intuitively grasp the movement of the main camera, enabling more natural shooting operations. become.

(Appendix B2) The operator system further comprises a third operation interface for moving the position of the main camera within a third plane that intersects the first plane and the second plane.
A remote camera system according to Appendix B1.

(Appendix B3) The remote camera system according to Appendix B1 or B2, wherein the second plane is perpendicular to the optical axis of the main camera.

(Appendix B4) The remote camera system according to any one of Appendices B1 to B3, wherein the first plane, the second plane, and the third plane are orthogonal to each other.

(Appendix B5) The operator system has a touch panel and a control unit,
The control unit
On the screen of the touch panel, display a first operation area functioning as the first operation interface, display a second operation area functioning as the second operation interface, and display a second operation area functioning as the third operation interface. Displaying a functioning third operation area,
The position of the main camera is moved within the first plane based on the movement of the operating body that approaches or contacts the first operating area, and the movement of the operating body that approaches or contacts the second operating area. moving the position of the main camera within the second plane based on the above, and moving the position of the main camera within the third plane based on the movement of the operating body that approaches or touches the third operation area The remote camera system of Appendix B2.

(Appendix B6) The operator system has a display unit, a controller having operation keys, and a control unit,
The control unit
causing the display unit to display a first operation area functioning as the first operation interface, displaying a second operation area functioning as the second operation interface, and functioning as the third operation interface displaying a third operation area and displaying an operation object that moves on the screen in accordance with an operation input to the operation key;
The position of the main camera is moved within the first plane based on the movement of the operating body in the first operation area, and the position of the main camera is moved based on the movement of the operating body in the second operation area. The remote camera system according to appendix B2, wherein the position of the main camera is moved within the third plane based on the movement of the operating body in the third operation area, and the position of the main camera is moved within the second plane. .

(Appendix B7) The operator system has a display unit, a controller having operation keys and operation buttons, and a control unit,
The control unit
An operation of causing the display unit to display operation areas functioning as the first operation interface, the second operation interface, and the third operation interface, and moving on the screen according to the operation input to the operation keys. show the body
switching the function of the operation area according to the operation of the operation button;
when the function of the operation area is the first function, moving the position of the main camera within the first plane based on the movement of the operating body in the operation area;
when the function of the operation area is the second function, moving the position of the main camera within the second plane based on the movement of the operating body in the operation area;
The remote camera system according to appendix B2, wherein when the function of the operation area is the third function, the position of the main camera is moved within the third plane based on the movement of the operating body in the operation area.

(Appendix B8) The operator system has a controller having a motion sensor and an operation button, and a control unit,
The control unit
A first mode in which the controller functions as the first operation interface, the second operation interface, or the third operation interface according to the pressing state of the operation button, and the first mode switching between different second modes;
In the first mode, a motion component corresponding to motion in a plane perpendicular to an axis preset in the controller is extracted from the motion information output from the motion sensor, and the extracted motion component is moving the position of the main camera within the first plane, within the second plane, or within the third plane based on
The remote camera system according to appendix B2, wherein in the second mode, the position of the main camera is three-dimensionally moved based on motion information output from the motion sensor.

(Appendix B9) The remote camera system according to Appendix B8, wherein the control unit switches the function in the first mode according to the operation of the operation button.

(Appendix B10) An operator system used to operate a camera unit having a main camera and a motion unit for controlling the position of the main camera,
a first operation interface for moving the position of the main camera within a first plane including the optical axis of the main camera; and a position of the main camera within a second plane intersecting the first plane. a second operation interface for moving,
operator system.

5 remote camera system 10a, 10b camera unit 11 main camera 12 eye view camera 13 motion unit 13a robot arm 13b base 14 control box 20a, 20b rear view camera 30a, 30b operator system 31 operation section 32 display section 33 control section 34 sensor section 35 Storage unit 35a Program 36 Audio input unit 37 Audio output unit 40 Control system 41 Unit control unit 42 Main video output unit 43 Eye view providing unit 44 Rear view providing unit 45 Communication tool providing unit 46 Camera work execution unit 46a Preset execution unit 46b Scenario execution Unit 46c Real-time operation unit 47 Authority management unit 48 Storage unit 48a TCP setting information 48b Preset information 48c Scenario information 48d Operation authority information 48e Operation log 48f Communication log

Claims (13)

a first movement of rotating a coordinate system having a main camera and a reference point connected to the main camera at one end and having a known positional relationship with the one end as an origin; and a second movement of moving the reference point. and a motion unit capable of two movements; and a representative point set on the imaging element of the main camera and the reference point are matched to control the motion unit to perform the first movement. a control system that changes the pose of the main camera by the second movement and changes the position of the main camera by the second movement;
remote camera system. 2. The control system according to claim 1, wherein the control system holds point setting information indicating a positional relationship between the representative point set on the imaging element and the one end of the motion unit for each type of the main camera. remote camera system. 3. The remote camera system according to claim 1, wherein the representative point is an intersection point between the optical axis of the main camera and the imaging device. The control system holds preset information including location information indicating the position of the camera unit at the shooting site, position information indicating the position of the reference point and the rotation angle of the coordinate system, and setting information of the main camera. 4. The remote camera system according to any one of claims 1 to 3, wherein the camera unit is controlled based on the preset information according to an operation by an operator. When the control system receives an instruction to register preset information from the operator, the control system outputs a thumbnail of the image output from the main camera at the time of the registration instruction, the location information at the time of the registration instruction, and the position information. , and the setting information, and holds the generated thumbnail, the location information, the position information, and the setting information as the preset information. The remote camera system according to any one of claims 1 to 5, wherein the control system controls the motion unit to move the reference point along an arc of predetermined radius centered on a set point. The remote camera system includes an operator system used to operate the camera unit,
The operator system comprises a first operation interface for moving the reference point within a first plane including the optical axis of the main camera, and a second plane intersecting with the first plane. 7. The remote camera system according to any one of claims 1 to 6, further comprising a second operation interface for moving the . 8. The remote camera system according to claim 7, wherein said operator system further comprises a third operation interface for moving said reference point within a third plane intersecting said first plane and said second plane. The camera unit further has an eye view camera that outputs an omnidirectional video,
The operator system further includes a sensor unit that detects the line-of-sight direction of the operator,
The control system extracts an image of a predetermined range based on the line-of-sight direction of the operator from the omnidirectional image output from the eye view camera based on the output of the sensor unit, and transmits the extracted image to the operator system. 9. The remote camera system according to claim 7 or 8, wherein the transmission to the The remote camera system includes a plurality of rear-view cameras installed at positions where the camera unit can be photographed from behind, and an operator system used to operate the camera unit,
When the camera unit is controlled based on the preset information, the control system determines whether the camera unit among the plurality of rear-view cameras based on the position of the camera unit indicated by the location information in the preset information. 6. The remote camera system according to claim 4 or 5, wherein a rear view camera included in an angle of view is selected and an output image of the selected rear view camera is transmitted to the operator system. a first movement of rotating a coordinate system having a main camera and a reference point connected to the main camera at one end and having a known positional relationship with the one end as an origin; and a second movement of moving the reference point. a unit control unit that controls a camera unit having a motion unit capable of two movements,
The unit control unit aligns a representative point set on the imaging device of the main camera with the reference point, controls the motion unit, and changes the posture of the main camera by the first movement, changing the position of the main camera with the second movement;
control system. a first movement of rotating a coordinate system with a main camera and a reference point connected at one end to the main camera and having a known positional relationship with the one end by a computer; controlling a camera unit having a motion unit capable of a second movement to move;
In the controlling step, the computer aligns a representative point set on the imaging device of the main camera with the reference point, controls the motion unit, and moves the main camera by the first movement. changing posture and changing the position of the main camera with the second movement;
control method. a first movement for rotating a coordinate system having a main camera and a reference point connected at one end to the main camera and having a known positional relationship with the one end as an origin; executing a process of controlling a camera unit having a motion unit capable of a second movement to be moved;
In the processing, the computer matches a representative point set on the imaging device of the main camera with the reference point, controls the motion unit, and adjusts the posture of the main camera by the first movement. and changing the position of the main camera with the second movement;
control program.

Images