JP5998829B2 - Camera work control device and camera work control program - Google Patents

Description

  The present invention relates to a camera work control device and a camera work control program.

  For the purpose of generating a content image (universal content image) optimized for the viewing environment of the viewer, the sensibility effect (for example, the feeling of power of the image) given to the viewer by the moving image has been investigated. For example, the image generation device uses a VR (Virtual Reality) feature that camera work can be freely changed, and generates images with different motions from the same original camera work. The emotional effect (subjective impression) is evaluated by a subjective evaluation method in which an evaluator presents images with different movements to the viewer and the viewer reports the strength of the impression received by the viewer. Patent Document 1 discloses an image generation apparatus for generating a moving image according to a viewer's preference.

JP 2008-262392 A

  However, the image generation apparatus disclosed in Patent Document 1 needs to create camerawork additional information for emphasizing comfort and enjoyment in advance through human operation, such as comfort and enjoyment. There is a problem in that a moving image that gives the viewer an impression of the above cannot be generated without human intervention. Therefore, application of the image generation device disclosed in Patent Document 1 to an interactive VR in which a viewer operates the camera by himself / herself is limited.

  The present invention has been made in view of the above points, and a camera work control device capable of generating a moving image that gives an impression of comfort and enjoyment to a viewer without human intervention, And a camera work control program.

  According to one aspect of the present invention, a calculation unit that calculates a distance distribution of an output image for each unit region predetermined in an output image based on the input image, and the output image as a moving image according to the distance distribution And a controller for controlling the camera work in the camera work control apparatus.

  One embodiment of the present invention is the camera work control apparatus, wherein the calculation unit calculates the distance distribution based on distance information associated with the input image.

  One embodiment of the present invention is the camera work control device, wherein the calculation unit calculates the distance distribution based on distance information calculated from computer graphics as the input image.

  One embodiment of the present invention is the camera work control device, wherein the control unit controls a camera angle as the camera work.

  One embodiment of the present invention is the camera work control device, wherein the control unit controls the camera work based on an optical flow corresponding to the distance distribution.

  One embodiment of the present invention is the camera work control apparatus, wherein the control unit controls the camera work based on a gradient of the optical flow.

  Further, according to one aspect of the present invention, the calculation unit calculates a distribution of the degree of interest for each subject in the output image according to the degree of interest for each subject predetermined in the input image, and the control unit Is a camera work control apparatus that controls the camera work based on a distribution of the degree of interest for each subject in the output image.

  One embodiment of the present invention is a camera work control apparatus in which the control unit controls a camera path as the camera work.

  Further, according to one embodiment of the present invention, a computer calculates, as a moving image, a procedure for calculating a distance distribution of an output image for each unit region predetermined in an output image based on the input image, and the distance distribution. And a procedure for controlling camera work in the output image.

  According to the present invention, the control unit controls the camera work in the output image as the moving image according to the distance distribution. Thereby, the camera work control device and the camera work control program can generate a moving image that gives the viewer an impression of comfort and enjoyment without the intervention of a human operation.

It is a block diagram which shows the structural example of the camera work control apparatus in one Embodiment of this invention. It is a figure which shows the example of a computer graphics and a depth image in one Embodiment of this invention. It is a figure for demonstrating perspective projection conversion in one Embodiment of this invention. It is a figure which shows the example of the camera path | pass in one Embodiment of this invention. It is a figure which shows the example of the output image which shows the optical flow in one Embodiment of this invention. It is a figure which shows the area | region predetermined in the output image in one Embodiment of this invention. It is a figure which shows the example of a relationship between a linear camera path | pass and an optical flow in one Embodiment of this invention. It is a figure which shows the example of a relationship between the camera path of a curve, and an optical flow in one Embodiment of this invention. It is a figure which shows the example of the camera angle in one Embodiment of this invention. It is a figure which shows the example of distribution of the interest degree for every to-be-photographed object in one Embodiment of this invention. It is a figure which shows the example of a change of the interest degree according to the position of the camera in one Embodiment of this invention. It is a figure which shows the example of the end point of a camera path | pass in one Embodiment of this invention. It is a figure which shows the example of a relationship between the optical flow and threshold value in one Embodiment of this invention. It is a figure which shows the example of the camera angle in the end point in one Embodiment of this invention. It is a flowchart which shows the example of the operation | movement procedure which determines the end point of a camera path in one Embodiment of this invention. It is a flowchart which shows the example of the operation | movement procedure which defines the straight camera path in one Embodiment of this invention. It is a flowchart which shows the example of the operation | movement procedure which calculates the optical flow in one Embodiment of this invention. It is a flowchart which shows the example of the operation | movement procedure which calculates distribution of an interest degree in one Embodiment of this invention. It is a flowchart which shows the example of the operation | movement procedure which controls camera work in one Embodiment of this invention.

  An embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration example of a camera work control apparatus. The camera work control device 100 generates a moving image that gives the viewer an impression of comfort and enjoyment. The camerawork control device 100 includes a calculation unit 110, a storage unit 120, an operation unit 130, a control unit 140, and a display unit 150.

  The calculation unit 110 reads, from the storage unit 120, three-dimensional space data (scene data) indicating computer graphics (CG) as an input image. The calculation unit 110 calculates the distance distribution of the output image based on the input image when the camera moves on the camera path (movement path) defined in the virtual three-dimensional space. Further, the calculation unit 110 calculates the optical flow of the output image based on the input image based on the distance distribution of the output image based on the input image. Note that the distance distribution and the optical flow have a reciprocal relationship.

  Here, the optical flow is a vector indicating a motion distribution in an image. When the input image is a real image, the calculation unit 110 performs an optical flow of the output image based on the input image based on a difference between frames included in the input image as a moving image by image processing such as a block matching method. calculate. On the other hand, when the input image is computer graphics, since all geometric information representing the three-dimensional space of the input image is known, the calculation unit 110 is included in the three-dimensional space data without performing image processing. The optical flow of the output image based on the input image is calculated based on the computer graphics model and the numerical data regarding the camera work.

  In addition, when the input image is computer graphics, the calculation unit 110 performs a subject virtually placed in the three-dimensional space of the input image from a camera (viewpoint) virtually placed in the three-dimensional space of the input image. A depth (depth) image indicating the distance to the surface of the (object) is generated. For example, when OpenGL (registered trademark) is used, the calculation unit 110 calls a glReadPixels function (glReadPixels (0, 0, nWidth, nHeight, GL_DEPTTH_COMPONENT, GL_FLOAT, depthImage)) in the process of rendering. Get.

  FIG. 2 shows examples of computer graphics and depth images. In FIG. 2A, computer graphics as an input image is shown in the screen coordinate system. In this computer graphics, a temple is depicted as an example of the subject 200. FIG. 2B is a depth image generated based on the computer graphics. In FIG. 2B, the shorter the distance from the camera (viewpoint) to the surface of the subject 200 is, the more black it is, and the longer it is, the white it is.

  FIG. 3 is a diagram for explaining perspective projection conversion. When OpenGL (registered trademark) is used, the calculation unit 110 can set perspective projection conversion for converting the view coordinate system to the screen coordinate system for the three-dimensional space of the input image by calling the glFrustum function. . The calculation unit 110 converts the Z coordinate value (depth) from the front clip surface 310 to the rear clip surface 320 into a range of −1.0 to 1.0 based on Expression (1).

  Here, near indicates the distance from the camera 300 to the front clip surface 310. Further, far indicates the distance from the camera 300 to the rear clip surface 320. Further, top indicates the upper end (Y-axis positive direction) of the front clip surface 310. “Bottom” indicates a lower end (Y-axis negative direction) of the front clip surface 310. Further, right indicates the right (X-axis positive direction) end of the front clip surface 310. Left indicates the left (X-axis negative direction) end of the front clip surface 310.

  When the depth image is read out in the GL_FLOAT format, the value obtained by perspective projection conversion to the range of −1.0 to 1.0 is further normalized to the range of 0.0 to 1.0. Therefore, the calculation unit 110 calculates the distance Z from the camera 300 (viewpoint) to the surface point of the subject for each pixel of the input image by performing this inverse operation on the depth image.

  The calculation unit 110 also includes a viewpoint movement speed (Vx, Vy, Vz), a viewpoint rotation angular speed (ωp, ωy, ωr), a screen coordinate (Sx, Sy), and a viewing volume (right) in the view coordinate system. , Left, top, bottom, far, and near), a motion vector (Fx, Fy) in the screen coordinate system is calculated. Here, the screen coordinates (Sx, Sy) may be normalized in a range of −1.0 to 1.0.

  The motion vector in the screen coordinate system of the input image is generated when the camera 300 (see FIG. 3) moves in the three-dimensional space of the input image. The motion vector is calculated by the equations (2) to (6) according to the factors that occur. Hereinafter, the sum of these motion vectors is referred to as “optical flow”. In addition, the calculation unit 110 calculates an optical flow for each unit region predetermined in the output image based on the input image.

  A motion vector in the screen coordinate system of the input image, which is generated when the camera 300 moves parallel to the subject in the three-dimensional space of the input image, is expressed by Expression (2). Here, D indicates the subject distance. Thus, the distance distribution and the optical flow have a reciprocal relationship. In the view coordinate system, D = −Z.

  In addition, a motion vector in the screen coordinate system of the input image, which is generated when the camera 300 moves linearly in the three-dimensional space of the input image, is expressed by Expression (3). Here, D indicates the subject distance. In the view coordinate system, D = −Z.

 Further, a motion vector in the screen coordinate system of the input image, which is generated by the yaw rotation (left-right rotation) of the camera 300 in the three-dimensional space of the input image, is expressed by Expression (4).

 In addition, a motion vector in the screen coordinate system of the input image, which is generated when the camera 300 performs pitch rotation (up and down rotation) in the three-dimensional space of the input image, is expressed by Expression (5).

 In addition, a motion vector in the screen coordinate system of the input image, which is generated when the camera 300 rolls (twists) in the three-dimensional space of the input image, is expressed by Expression (6).

  Further, the calculation unit 110 calculates the distribution of the interest level for each subject 200 in the output image according to the interest level for each subject predetermined in the three-dimensional space of the input image. Details of the distribution of interest will be described later with reference to FIG.

  FIG. 4 shows an example of a camera path. In FIG. 4, the camera path 400 is a curved moving path (shown by a solid line) as an example with respect to a straight moving path (shown by a broken line) toward the target position 700 designated by the user. It has been established.

  FIG. 5 shows an example of an output image showing an optical flow. FIG. 5A illustrates a case where the camera 300 (see FIG. 3) is turning around the subject 200 at high speed along the camera path 400 (see FIG. 4) while facing the subject 200. An example of optical flow is indicated by the direction and size of the arrow.

  In FIG. 5 (A), the lower region of the output image (corresponding to the region where the ground is drawn in FIG. 2) shows an optical flow of a predetermined value or more that faces rightward (X-axis positive direction). ing. Further, an optical flow equal to or less than the predetermined value is shown in the central area of the output image (corresponding to the area where the subject 200 is drawn in FIG. 2). In addition, in the upper area of the output image (corresponding to the area in which the sky is drawn in FIG. 2), an optical flow facing the left direction (X-axis negative direction) is shown.

  On the other hand, FIG. 5B shows an example of an optical flow in the case where the camera 300 is turning at low speed around the subject 200 along the camera path 400 while facing the subject 200. It is indicated by size. In FIG. 5B, an optical flow concentrated toward one point is shown in the lower right area of the output image (corresponding to the area where the right outline of the subject 200 is drawn in FIG. 2). Yes. Here, the reason why the optical flow concentrated toward one point is shown is that the camera 300 starts to move away from the right contour of the subject 200 (see FIG. 2) as the camera 300 turns. .

Returning to FIG. 1, the description of the configuration example of the image generation apparatus will be continued.
The storage unit 120 stores in advance three-dimensional spatial data (scene data) indicating computer graphics as an input image. In addition, the storage unit 120 stores a calculation result by the calculation unit 110, for example, information indicating an optical flow. In addition, the storage unit 120 stores numerical data related to camera work such as the camera path 400 (see FIG. 4) in advance. Note that the storage unit 120 may store a program for operating the control unit 140 in advance.

  The operation unit 130 receives an operation input by the user and outputs a signal corresponding to the operation input to the control unit 140. Here, the signal corresponding to the operation input is, for example, a signal indicating various setting conditions (threshold value or the like).

  The control unit 140 controls camera work in the output image according to the optical flow of the output image as a moving image. Here, the control unit 140 controls at least one of camera angle, camera path, and zoom magnification as camera work. Details of the control unit 140 will be described later.

  The display unit 150 displays a display area defined in the output image among output images as moving images generated by camera work controlled by the control unit 140. Further, the display unit 150 displays an operation panel (not shown) for the user to operate the operation unit 130.

Next, details of the output image will be described.
FIG. 6 shows a predetermined area in the output image. The output image is an image seen from the camera 300 (see FIG. 3) placed in the three-dimensional space of the input image. In FIG. 6, the output image is divided into 16 × 16 unit areas 500-XY (unit areas 500-0-0 to unit areas 500-15-15). Hereinafter, items common to all of the unit areas 500-XY are denoted as “unit area 500” by omitting the symbols X and Y. The boundary line of the unit area 500 may not be displayed on the output image.

  Hereinafter, the unit area 500-0-0, the unit area 500-0-15, the unit area 500-7-15, and the rectangular area that apexes the unit area 500-7-0, that is, the left of the output image The half area is called “left area”. In addition, the unit area 500-8-0, the unit area 500-8-15, the unit area 500-15-15, and the rectangular area that apexes the unit area 500-15-0, that is, the right half of the output image The area is referred to as a “right area”.

  In addition, hereinafter, the unit area 500-0-7, the unit area 500-0-15, the unit area 500-15-15, and the rectangular area that apexes the unit area 500-15-8, that is, above the output image Half of the area is called the “upper area”. In addition, the unit area 500-0-0, the unit area 500-0-7, the unit area 500-15-7, and the rectangular area that apexes the unit area 500-15-0, that is, the lower half of the output image The area is referred to as a “lower area”.

  A coefficient may be defined for each unit region 500. Here, for example, the coefficient determined in the unit region 500 at a predetermined position (for example, near the center) of the output image may be set to have a relatively large weight compared to other positions.

Next, details of the control unit will be described.
FIG. 7 shows an example of the relationship between a straight camera path and an optical flow. FIG. 7A shows a subject 200 and a subject 210 that are arranged in advance in the three-dimensional space of the input image. Here, it is assumed that the subject 210 is arranged in advance on the right side of the straight camera path 410 (initial camera path) toward the subject 200. On the straight camera path 410, as an example, a control point 411, a control point 412 and a control point 413 are arranged.

  FIG. 7B shows the optical flow of the output image for each frame of the output image, that is, for each position on the straight camera path 410. The horizontal axis indicates the frame of the output image. The vertical axis represents the optical flow of the output image. Here, for the optical flow, the sum in the right region, the sum in the left region, the sum in the upper region, and the sum in the lower region are shown. Note that when the sum of the optical flows is calculated, the optical flows may be weighted for each unit region 500 by a coefficient.

  Since the subject 210 is arranged in advance on the right side of the straight camera path 410, in FIG. 7B, the absolute value of the sum of the optical flows in the right region of the output image is the optical flow in the left region of the output image. It is shown that it is larger than the absolute value of the sum.

  Since the camera path 410 is a straight line, the optical flow of the output image can be calculated only from Equation (3). That is, the optical flow of the output image is proportional to the moving speed of the camera 300 (see FIG. 3) and the screen coordinates, and is inversely proportional to the distance, so that it can be calculated approximately based only on the distance distribution of the output image. .

  FIG. 8 shows an example of the relationship between a curved camera path and an optical flow. When the position of each control point on the straight camera path 410 is moved by the control unit 140, the camera path 410 is modified from a straight line to a curved line. FIG. 8A shows a subject 200 and a subject 210 that are arranged in advance in the three-dimensional space of the input image. Here, the subject 200 and the subject 210 are arranged in advance at the same position as that shown in FIG. Note that the straight camera path shown in FIG. 7A is indicated by a broken line in FIG.

  The position of each control point on the straight camera path shown in FIG. 7A is moved by the control unit 140 and connected by spline interpolation. As a result, the camera path 410 is modified from a straight line to a curved line as shown in FIG. In FIG. 8A, each control point is moved in a direction orthogonal to the straight camera path 410 shown in FIG. In addition, the control unit 140 uses the camera path so that the sum of the optical flow sum in the right region of the output image and the sum of the optical flow in the left region of the output image approaches 0. Bend 410. Since a sufficient distance from the subject 210 is maintained and the camera path 410 is bent, the output image can give the viewer a comfortable visual effect without excessive stimulation.

  FIG. 8B shows an optical flow of the output image for each frame of the output image, that is, for each position on the camera path 410 of the curve. The horizontal axis indicates the frame of the output image. The vertical axis represents the optical flow of the output image. Here, for the optical flow, the sum in the right region, the sum in the left region, the sum in the upper region, and the sum in the lower region are shown.

  Since the subject 210 is arranged in advance on the right side of the curved camera path 410 toward the subject 200, in FIG. 8B, the absolute value of the sum of the optical flows in the right region of the output image is the left region of the output image. It is shown that it is larger than the absolute value of the sum of the optical flows at. Further, since the camera path 410 is bent by the control unit 140, in FIG. 8B, the sum of the optical flow sum in the right region of the output image and the sum of the optical flows in the left region of the output image is shown in FIG. Compared to the case shown in (B), it is shown that the value is close to 0.

  It should be noted that the control unit 140 prevents the camera path 410 from colliding with another subject (not shown) due to the camera path 410 being bent, that is, maintains the minimum distance between the other subject and the camera 300. In addition, the position of the camera path 410 may be modified. For the upper area and lower area of the output image, the control unit 140 causes the sum of the optical flow in the upper area of the output image and the sum of the optical flows in the lower area of the output image to approach 0. The camera path 410 may be bent.

  In addition, the camera path 410 is bent so that the sum of optical flows in the left region and the right region, or the upper region and the lower region is increased, so that the output image has an impression such as moderate comfort and fun. Can give the viewer the opposite visual effect (eg, excessive speed). In addition, the output image is manipulated by adjusting the bending amount of the camera path 410 according to the adjustment factor, so that an intermediate visual effect (for example, moderate speed feeling) that gives an impression of appropriate comfort and enjoyment can be viewed. Can be given to a person.

  FIG. 9 shows an example of the camera angle. The control unit 140 controls the camera angle (rotation angle with respect to the camera path) of the camera 300 (see FIG. 3) based on the optical flow of the output image. 9A and 9B, for comparison, an example of the camera angle of the camera 300 moving on the straight camera path 410 is shown by the direction of the arrow at the control point 411, the control point 412, and the control point 413. It is shown.

  In FIG. 9A, the camera angle of the camera 300 is not changed according to the optical flow of the output image. On the other hand, in FIG. 9B, the camera angle of the camera 300 is changed according to the optical flow of the output image. Specifically, at the control point 411, since the camera 300 passes in the vicinity of the subject 210, the camera 300 is directed in the direction in which the optical flow of the output image is large, that is, the direction of the subject 210. At the control point 412, since the camera 300 passes through the subject 210, the gradient of the optical flow becomes small. For this reason, the camera 300 is directed in the direction in which the optical flow of the output image is large, that is, in the middle of the direction of the subject 210 and the direction of the subject 200 (the direction in which the amount of rotation of the camera angle decreases). At the control point 413, since the camera 300 is approaching the subject 200, the gradient of the optical flow becomes sufficiently small. Therefore, the camera 300 is directed toward the subject 200 without changing the camera angle.

  As a result, the camera angle of the camera 300 is determined in a region where the viewer is likely to be interested, that is, a region where the optical flow of the output image is large, or a region where a subject (object) exists. The viewer can be provided with a comfortable visual effect that can satisfy the viewer's interest. If the camera angle of the camera 300 is determined in a direction in which the optical flow becomes smaller, the output image has a visual effect (for example, bored feeling) opposite to an impression such as moderate comfort and fun. Can be given to the viewer. Further, the output image has an intermediate visual effect (for example, a moderate sense of speed) that gives an impression of appropriate comfort and enjoyment as the amount of rotation of the camera angle of the camera 300 is manipulated by an adjustment coefficient. Can be given to the viewer.

  FIG. 10 shows an example of the distribution of the degree of interest for each subject. Information indicating a predetermined degree of interest is added as metadata to each subject (object) included in the three-dimensional space data indicating computer graphics as an input image. Here, the degree of interest is the degree to which the viewer is interested in the subject, and is determined in advance by, for example, the creator who created the three-dimensional space data.

  For example, assuming that the degree of interest in a subject is proportional to the image feature amount (for example, luminance, color) of the subject, the subject is rendered according to the image feature amount, and thus the image feature amount in the output image The distribution is a distribution of interest in the output image. In FIG. 10, since the luminance of the subject 200 mainly drawn at the center of the left region of the output image is high, the degree of interest for each subject is high at the center of the left region of the output image.

  FIG. 11 shows an example of a change in the degree of interest according to the position of the camera. The horizontal axis indicates the position of the camera 300. In FIG. 11, the control point 411, the control point 412, and the control point 413 illustrated in FIG. 9A are illustrated as the positions of the camera 300. The vertical axis represents the sum of the degrees of interest in a predetermined area of the output image. Here, for the degree of interest, as an example, the sum of the degrees of interest in the right region and the sum of the degrees of interest in the left region are shown. Note that the degree of interest may be stored as a pixel value of the output image.

  The camera angle of the camera 300 is changed according to the difference in the interest distribution for each predetermined region in the output image. That is, the camera angle of the camera 300 is determined by the control unit 140 in the direction that the viewer wants to see. Note that, when the difference in the interest distribution is calculated, each unit region 500 may be weighted with a coefficient.

  At the control point 411 shown in FIG. 9A, the camera 300 facing the subject 200 passes near the subject 210. Therefore, at the control point 411 shown in FIG. , It is higher than the sum of interest in the left area. Therefore, as shown in FIG. 9B, at the control point 411, the camera 300 is directed rightward where the sum of the degrees of interest is high.

  Further, since the camera 300 has passed the subject 210 at the control point 412 shown in FIG. 9A, at the control point 412 shown in FIG. 11, the sum of the interest degrees in the right region and the sum of the interest degrees in the left region. Is almost the same. Therefore, as shown in FIG. 9B, at the control point 412, the camera 300 is directed in substantially the same direction as the camera path 410. The same applies to the control point 413.

  For the upper and lower regions of the output image, the control unit 140 determines whether the viewer has a difference between the total interest level in the upper region of the output image and the total interest level in the lower region of the output image. You may point the camera 300 in the upward direction or the downward direction that you want to see. As a result, the camera angle of the camera 300 is determined in a region where the viewer's interest is easily attracted, that is, a region where the sum of the degrees of interest in the output image is large, so that the output image can satisfy the viewer's interest. A comfortable visual effect can be given to the viewer. If the camera angle of the camera 300 is determined in a direction in which the sum of the degrees of interest in the left region and the right region, or the upper region and the lower region is reduced, the output image has moderate comfort and enjoyment. It is possible to give the viewer a visual effect (for example, bored feeling) that is opposite to the impression. Further, the output image has an intermediate visual effect (for example, a moderate sense of speed) that gives an impression of appropriate comfort and enjoyment as the amount of rotation of the camera angle of the camera 300 is manipulated by an adjustment coefficient. Can be given to the viewer.

  FIG. 12 shows an example of the end point of the camera path. The control unit 140 determines an end point 410a as a position suitable for the viewer to view the subject 200 before the target position 700 designated by the user. Here, the control unit 140 determines the distance (observation distance) from the target position 700 to the end point 410a based on the optical flow and a predetermined threshold value. Accordingly, the camera 300 (see FIG. 3) can capture the subject 200 as an output image regardless of the size of the subject 200 from the viewpoint from the end point 410a.

  FIG. 13 shows an example of the relationship between the optical flow and the threshold value. FIG. 13 shows the optical flow of the output image for each frame of the output image, that is, for each position on the camera path 410 (see FIG. 12). The horizontal axis indicates the frame of the output image. The vertical axis represents the optical flow of the output image. Here, for the optical flow, the sum in the right region, the sum in the left region, the sum in the upper region, and the sum in the lower region are shown.

  As the camera 300 (see FIG. 3) moving along the camera path 410 (see FIG. 12) approaches the subject 200, the absolute value of the optical flow of the output image increases. In addition, when the absolute value of the optical flow is too large, that is, when the camera 300 and the subject 200 are too close, the camera 300 exceeds a position suitable for the viewer to view the subject 200. become. Therefore, the control unit 140 determines the end point 410a of the camera path 410 at a position where the optical flow of the output image becomes equal to a predetermined threshold value. In FIG. 13, the end point 410a of the camera path 410 is determined at a position where the optical flow in the right region of the output image once falls below the threshold value (shown by a broken line) and becomes equal again.

  FIG. 14 shows an example of the camera angle at the end point. In FIG. 14A, since the camera path 410 is defined in a locus that approaches the subject 200 from an oblique direction, the camera angle 414 from the camera 300 (see FIG. 3) at the end point 410a of the camera path 410 is determined. Since the visible subject 200 does not face the camera 300, the composition is not suitable for viewing in the output image.

  In FIG. 14B, the control unit 140 compares the optical flow in the left region of the output image with the optical flow in the right region of the output image for the end point 410a of the camera path 410, and generates a larger optical flow. The camera 300 is directed toward the right region. The control unit 140 also corrects the coordinates of the end point 410a of the camera path 410. Here, the control unit 140 corrects the coordinates of the end point 410a of the camera path 410 so that the camera path 410 is parallel to the subject 200, for example.

Next, an example of the operation procedure of the camera work control device will be described.
FIG. 15 is a flowchart illustrating an example of an operation procedure for determining the end point of the camera path.
(Step Sa1) It is assumed that the user designates the target position 700, which is the movement destination of the camera 300, on the output image by operating the operation unit 130 (for example, clicking with a mouse). The control unit 140 acquires the coordinates of the target position 700 in the screen coordinates (for example, the coordinates of the mouse pointer when the click operation is performed) based on the signal according to the operation input.

  (Step Sa2) The calculation unit 110 calculates the depth value of the target position 700 and causes the storage unit 120 to store the calculated depth value. The control unit 140 acquires the depth value calculated by the calculation unit 110 from the storage unit 120.

(Step Sa3) The control unit 140 converts the coordinates of the target position 700 in the screen coordinates into the coordinates of the target position 700 in the view coordinate system based on the depth value.
(Step Sa4) The control unit 140 converts the coordinates of the target position 700 converted into the view coordinate system into the coordinates of the target position 700 (target point) in the world coordinate system.

(Step Sa5) The control unit 140 determines the stop position of the camera 300 that moves on the camera path. Here, the control unit 140 determines the observation distance between the subject at the target position 700 and the camera 300 according to the size of the subject. For example, the control unit 140 determines the coordinates of the end point (target point) of the camera path so that the camera 300 stops when the optical flow exceeds a predetermined setting value C ₆ (end point reference flow). As the subject at the target position 700 increases, the optical flow increases more rapidly as the subject and the camera 300 approach each other.

(Step Sa6) The control unit 140 determines the camera angle at the end point of the camera path. Here, the control unit 140 determines the camera angle (rotation angle) at which the end point (target point) of the camera path is desired from the current position of the camera 300, the horizontal angle YY _END and the vertical angle of the camera angle at the end point of the camera path. Set as PP _END

(Step Sa7) The control unit 140 corrects the camera angle of the camera 300. Here, based on the optical flow imbalance amounts F _LR and F _TB immediately before the end point and the adjustment coefficient C ₇ (end point angle control amount), the control unit 140 corrects the camera angle correction amount at the end point “ “ΔYY _END = F _LR × C ₇ ” and “ΔPP _END = F _TB × C ₇ ” are calculated.

  (Step Sa8) The control unit 140 corrects the stop position of the camera 300. Here, the control unit 140 corrects the stop position of the camera 300 to a position where the end point (target point) of the camera path is captured in the output image with the corrected camera angle.

FIG. 16 is a flowchart illustrating an example of an operation procedure for determining a straight camera path.
(Step Sb1) The control unit 140 determines the total movement distance L of the camera 300 (the distance from the current position of the camera 300 to the end point (target point)), that is, the length of the straight camera path.
(Step Sb2) The control unit 140 determines the total movement time T _TOTAL of the camera 300 by dividing the total movement distance L of the camera 300 by the speed V. Here, when the total moving distance L is shorter or longer than the predetermined threshold, the control unit 140 prevents the total moving time T _{TOTAL from} becoming an extreme value, for example, the total moving time T _{TOTAL is set} to the shortest 3 seconds to the longest 20 seconds. You may adjust.

(Step Sb3) The control unit 140 determines the number N of control points to be arranged in the straight camera path as the initial camera path in accordance with the total movement time T _TOTAL . Here, when the number N of control points is small, the control unit 140 cannot finely control the camera path. On the other hand, when the number N of control points is large, the camera path becomes an unstable camera path with many fluctuations. Therefore, the control unit 140 adds at least three control points and one control point to the camera path every 2 seconds, for example, “N = T _TOTAL / 2 + 3”. Also good.

  (Step Sb4) The control unit 140 arranges control points on a straight camera path as an initial camera path. That is, the control unit 140 generates N control points so as to linearly interpolate the current position of the camera 300 and the coordinates of the end point (target point) of the camera path.

FIG. 17 is a flowchart illustrating an example of an operation procedure for calculating an optical flow.
(Step Sc1) The calculation unit 110 divides the output image into a grid pattern in the X direction N _X pieces and the Y direction N _Y pieces (see FIG. 6). Hereinafter, each unit region is expressed as G _XY (0 ≦ X <N _X , 0 ≦ Y <N _Y ).
(Step Sc2) The calculation unit 110 converts the X and Y components of the optical flow generated when the camera 300 advances the camera path into all unit regions G _XY (0 ≦ X <N _X , 0 ≦ Y <N _Y ).

(Step Sc3) The calculating unit 110 calculates the sum of the X components of the optical flow (the optical flow F _L in the left region of the output image) in the unit region G _XY (0 ≦ X <N _X / 2, 0 ≦ Y <N _Y ). Is calculated.
(Step Sc4) The calculation unit 110 calculates the X component of the optical flow in the unit region G _XY (N _X / 2 ≦ X <N _X , 0 ≦ Y <N _Y ) (optical flow F _R in the right region of the output image). Calculate the sum.

(Step Sc5) The calculating unit 110 calculates the sum of the Y components of the optical flow (optical flow F _B in the lower region of the output image) in the unit region G _XY (0 ≦ X <N _X , 0 ≦ Y <N _Y / 2). Is calculated.
(Step Sc6) The calculation unit 110 calculates the Y component of the optical flow in the unit region G _XY (0 ≦ X <N _X , N _Y / 2 ≦ Y <N _Y ) (optical flow F _T in the upper region of the output image). Calculate the sum.

(Step Sc7) The calculation unit 110 performs optical flows F _L (M), F _R (M), F _B (M), and F _T (M) at the interpolation position M for interpolating between control points on the camera path. Is stored in the storage unit 120.

FIG. 18 is a flowchart illustrating an example of an operation procedure for calculating the interest degree distribution.
(Step Sd1) The calculation unit 110 divides the output image into a grid pattern in the X direction N _X pieces and the Y direction N _Y pieces (see FIG. 6). Hereinafter, each unit region is expressed as G _XY (0 ≦ X <N _X , 0 ≦ Y <N _Y ).

(Step Sd2) The calculation unit 110 calculates the total sum of the pixel values (the degree of interest I _L in the left region of the output image) in the unit region G _XY (0 ≦ X <N _X / 2, 0 ≦ Y <N _Y ). .
(Step Sd3) The calculation unit 110 calculates the total sum of the pixel values (degree of interest I _R in the right region of the output image) in the unit region G _XY (N _X / 2 ≦ X <N _X , 0 ≦ Y <N _Y ). To do.

(Step Sd4) The calculation unit 110 calculates the sum of the pixel values (interesting degree I _B in the lower region of the output image) in the unit region G _XY (0 ≦ X <N _X , 0 ≦ Y <N _Y / 2). .
(Step Sd5) The calculation unit 110 calculates the sum of the pixel values (interest degree I _T in the upper region of the output image) in the unit region G _XY (0 ≦ X <N _X , N _Y / 2 ≦ Y <N _Y ). To do.

(Step Sd6) The calculation unit 110 calculates the interest levels I _L (M), I _R (M), I _B (M), and I _T (M) at the interpolation position M for interpolating between control points on the camera path. Is stored in the storage unit 120.

FIG. 19 is a flowchart illustrating an example of an operation procedure for controlling camera work.
(Step S <b> 1) The calculation unit 110 reads scene data from the storage unit 120.
(Step S2) The calculation unit 110 determines initial values of the position and orientation of the viewpoint of the camera 300 (see FIG. 3).
(Step S3) The calculation unit 110 performs rendering based on the scene data.

(Step S4) The control unit 140 determines an end point where the camera 300 moves. Here, for example, when the user clicks on the screen of the display unit 150 via the operation unit 130, the control unit 140 may determine the coordinates of the subject (object) displayed at the position as the end point. .
(Step S5) The control unit 140 generates a straight camera path 410 (see FIG. 7A). That is, the control unit 140 determines the coordinates of a straight camera path (initial camera path) connecting the current position of the camera 300 to the end point, and the interpolation curve for calculating the camera angle. Here, the control unit 140 calculates the number N of control points (for example, control points P _{0 to} P _N-1 ), the total movement time T _TOTAL of the camera 300 that moves the camera path at the speed V, and the camera angle. Determine. The interpolation position S for interpolating between control points on the camera path is referred to by a real value between an integer value 0 (current position of the camera 300) and an integer value (N−1) (end point).

(Step S6) The control unit 140 initializes an index value indicating the interpolation position S for interpolating between control points on the camera path to a value of 0.
(Step S <b> 7) The control unit 140 moves the camera 300 to the interpolation position S.
(Step S8) In order to obtain an RGB (red / green / blue) image and depth value (distance information) corresponding to the unit region 500, the calculation unit 110 performs off-screen rendering (trial drawing in a non-display state). Is executed in a reduced state. The pixel value of the RGB image is a value corresponding to the degree of interest predetermined as additional information for each computer graphics object (for example, the subject 200).

(Step S <b> 9) The calculation unit 110 calculates an optical flow and causes the storage unit 120 to store the calculated optical flow. Note that the calculation unit 110 may directly output the calculated optical flow to the control unit 140 without using the storage unit 120.
(Step S <b> 10) The calculation unit 110 calculates the interest level distribution in the output image and causes the storage unit 120 to store the calculated interest level distribution.

  (Step S11) The controller 140 moves the camera 300 to the interpolation position (S + ΔS). Here, in order to acquire the average value in the section including each control point, the control unit 140 calculates the interpolation position S for the intermediate point of the control points. When the control points are divided into K pieces, “ΔS = 1 / K”.

  (Step S12) The control unit 140 determines whether or not the index value S indicating the interpolation position is less than a value obtained by subtracting the value 1 from the number N of control points. When the index value S indicating the interpolation position is less than the value obtained by subtracting the value 1 from the number N of control points (step S12: Yes), the control unit 140 returns the process to step S7. On the other hand, when the index value S indicating the interpolation position is not less than the value obtained by subtracting the value 1 from the number N of control points (step S12: No), the control unit 140 advances the process to step S13.

(Step S13) The control unit 140 initializes an index value M indicating an interpolation position for interpolating between control points determined in the camera path to a value of 1. Thus, in the following step S14~ step 17, the control unit 140, the control point P ₀ in the initial position, the camera path (coordinate values of the control points) and camera angle (rotation angle of the camera at each control point) There is no adjustment. The control unit 140 adjusts the camera path and the camera angle for each control point between the control point P ₀ and the end point control point P _N−1 .

(Step S14) The control unit 140 outputs an output image area (see FIG. 6) for K frames (= interpolation position (M ± 0.5) range) before and after the camera path including the control point P _M. ) Calculate the sum of the average values of the optical flows for each. Specifically, the control unit 140 calculates the average value of optical flow F _L in the left area of the output image, the sum of the average value of optical flow F _R in the right region of the output image. The control unit 140 calculates the average value of optical flow F _T in the upper region of the output image, the sum of the average value of optical flow F _B in the area below the output image.

Thus, the control unit 140, the K frames over the front and rear, including a control point P _M of the focused (from the middle point between the control point immediately before, to the intermediate point between the control point immediately after), the optical flow Control using the average value is executed.

  If the motion of the image is symmetric in the upper and lower regions of the output image, the optical flow (motion vector) has the opposite value in the upper and lower regions, so the sum of the optical flows is an unbalanced amount. It becomes. The same applies to the left region and the right region. When the optical flow calculated for the camera path determined as the initial value includes a motion component other than the viewing direction of the camera 300 (see FIG. 3), the control unit 140 moves only in the viewing direction of the camera 300. Control is executed using the components.

(Step S15) The control unit 140 references the K frames (= interpolation position (M range ± 0.5)) over the front and rear of the camera path, the region (Figure 6 of an output image including the control point _{P M} ) Calculate the difference in the average value of the degree of interest for each. Specifically, the control unit 140 calculates the average value of the degree of interest I _L in the left area of the output image, the difference between the average value of the degree of interest I _R in the right area of the output image (imbalance). The control unit 140 calculates the average value of the degree of interest I _T in the upper region of the output image, the difference between the average value of the degree of interest I _B in the area below the output image (imbalance).

(Step S16) The control unit 140 to adjust the camera path to adjust the coordinates of the control points P _M, moving the control point P _M to adjust the position. The amount of horizontal movement in the X direction (see FIG. 3) is “(F _XL + F _XR ) × C ₀ ”. Also, the vertical movement amount in the Y direction (see FIG. 3) is “(F _YT + F _YB ) × C ₁ ”. Here, C ₀ is an adjustment coefficient defined as a control amount of the horizontal coordinate. Also, C ₁ is an adjustment factor defined as the control amount of the vertical coordinate.

(Step S17) The control unit 140 adjusts the camera angle at the control point _{P M.} The horizontal (yaw) rotation in the X direction (see FIG. 3) is “ΔYY _M = (F _XL + F _XR ) × C ₂ + (I _L −I _R ) × C ₃ ”. The vertical (pitch) rotation about the Y direction (see FIG. 3) is a _{"ΔPP M = (F YT + F} YB) × C 4 + (I T -I B) × C 5 ". Here, C ₂ is the adjustment factor defined as the control amount of the horizontal angle. Moreover, C ₄ is an adjustment factor defined as a control amount for the vertical angle. Also, C ₃ are adjustment factor defined as the control amount of the interest degree in the horizontal direction. Furthermore, C ₅ is a adjustment factor defined as the control amount of the interest degree in the vertical direction.

(Step S18) The control unit 140 adds a value 1 to an index value M indicating an interpolation position for interpolating between control points determined in the camera path.
(Step S19) The control unit 140 determines that the control unit 140 has an index value M indicating an interpolation position for interpolating between control points determined in the camera path, less than a value obtained by subtracting the value 2 from the number N of control points. It is determined whether or not. Thereby, in Step S14 to Step 17, the control unit 140 adjusts the camera path (the coordinate value of each control point) and the camera angle (the rotation angle of the camera at each control point) for the end point control point P _N-1. There is nothing to do. The control unit 140 adjusts the camera path and the camera angle for each control point between the control point P ₀ and the end point control point P _N−1 .

  When the index value M indicating the interpolation position for interpolating between control points determined in the camera path is less than the value obtained by subtracting the value 2 from the number N of control points (step S19: Yes), the control unit 140 performs step The process returns to S14. On the other hand, when the index value M indicating the interpolation position for interpolating between the control points determined in the camera path is not less than the value obtained by subtracting the value 2 from the number N of control points (step S19: No), the control unit 140 The process proceeds to step S20.

(Step S20) The controller 140 resets the timer variable T to the value 0.
(Step S <b> 21) The control unit 140 starts measuring time using the timer variable T.
(Step S22) The control unit 140 calculates the position and camera angle (rotation angle) of the camera 300 at time T, and updates the output image based on the calculated position and camera angle.

(Step S23) The calculation unit 110 executes rendering based on the updated output image.
(Step S24) The controller 140 substitutes the current time for the timer variable T.
(Step S <_b> 25) The control unit 140 determines whether the timer variable T is less than the total movement time T _TOTAL of the camera 300. When the timer variable T is less than the total movement time T _TOTAL of the camera 300 (step S25: Yes), the control unit 140 returns the process to step S22. On the other hand, when the timer variable T is not less than the total movement time T _TOTAL of the camera 300 (step S25: No), the control unit 140 advances the process to step S26.

  (Step S26) The control unit 140 determines whether or not to continue the operation based on a signal corresponding to the operation input. When the operation is continued (step S26: Yes), the control unit 140 returns the process to step S4. On the other hand, when the operation is not continued (step S26: No), the control unit 140 ends the process of controlling the camera work.

As described above, the camera work control device 100 performs the distance distribution of the output image (for example, the reciprocal of the distance distribution) for each unit region 500 predetermined in the output image based on the input image (see, for example, FIG. 6). For the optical flow in the relationship, for example, a calculation unit 110 that calculates (see FIG. 5) and a control unit 140 that controls camerawork in the output image as a moving image according to the distance distribution. .
Further, the camera work control program causes the computer to generate a moving image according to a procedure for calculating the distance distribution of the output image for each unit region 500 predetermined in the output image based on the input image, and the distance distribution. And a procedure for controlling camera work in the output image.

  With this configuration, the control unit 140 controls camera work in the output image as a moving image according to the distance distribution. Thereby, the camera work control apparatus 100 and the camera work control program can generate a moving image that gives the viewer an impression of comfort and enjoyment.

Further, the calculation unit 110 calculates the distance distribution (for example, a depth image) based on distance information associated with the input image.
The calculation unit 110 calculates the distance distribution based on distance information calculated from computer graphics as the input image.

Further, the control unit 140 controls a camera angle (see, for example, FIG. 9B) as the camera work.
The control unit 140 controls the camera work based on an optical flow corresponding to the distance distribution.
The control unit 140 controls the camera work based on the gradient of the optical flow (see, for example, FIG. 5). For example, the control unit 140 may control the camera work so that the sum of the optical flows of the left region and the right region in the output image becomes small.
In addition, the calculation unit 110 calculates a distribution of the degree of interest for each subject in the output image (see, for example, FIG. 10) according to the degree of interest for each subject determined in advance in the input image. 140 controls the camera work (for example, camera angle) based on the distribution of the degree of interest for each subject in the output image (for example, see FIG. 11).
Further, the control unit 140 controls a camera path (see, for example, FIG. 8A) as the camera work.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  For example, when the input image is a real image, the control unit 140 may generate an output image viewed from a viewpoint without the camera 300 based on a plurality of real images captured from multiple viewpoints.

  For example, the control unit 140 may change the camera work according to the viewing environment (for example, the screen size). When the screen size of the display unit 150 is small, the control unit 140 may change the camera work so that the visual effect is enhanced. In addition, when the screen size of the display unit 150 is large, the control unit 140 may change the camera work so that the visual effect is weakened. Further, the control unit 140 may change the camera work according to the individual difference (for example, age) of the viewer.

  Further, an execution process is performed by recording a program for realizing the camera work control device described above on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium. May be performed. Here, the “computer system” may include an OS and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

Further, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (Dynamic) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Random Access-memory)) is also included that holds a program for a certain period of time.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be for realizing a part of the functions described above.
Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 100 ... Camera work control apparatus, 110 ... Calculation part, 120 ... Memory | storage part, 130 ... Operation part, 140 ... Control part, 150 ... Display part, 200 ... Subject, 210 ... Subject, 300 ... Camera, 310 ... Front clip surface, 320 ... rear clip plane, 330 ... viewing frustum, 400 ... camera path, 410 ... camera path, 410a ... end point, 411 ... control point, 412 ... control point, 413 ... control point, 414 ... camera angle, 500 ... unit area 700 ... Destination position

Claims (9)

A calculation unit that calculates a distance distribution of the output image for each unit region predetermined in the output image based on the input image;
A control unit that controls camera work in the output image as a moving image according to the distance distribution;
A camera work control device comprising:   The camerawork control apparatus according to claim 1, wherein the calculation unit calculates the distance distribution based on distance information associated with the input image.   The camerawork control device according to claim 2, wherein the calculation unit calculates the distance distribution based on distance information calculated from computer graphics as the input image.   The camera control apparatus according to claim 1, wherein the control unit controls a camera angle as the camera work.   5. The camera work control apparatus according to claim 1, wherein the control unit controls the camera work based on an optical flow corresponding to the distance distribution. 6.   The camera work control apparatus according to claim 5, wherein the control unit controls the camera work based on a gradient of the optical flow. The calculation unit calculates a distribution of the degree of interest for each subject in the output image according to the degree of interest for each subject predetermined for the input image,
The camera work control according to any one of claims 1 to 6, wherein the control unit controls the camera work based on a distribution of the degree of interest for each subject in the output image. apparatus.   The camera control apparatus according to claim 1, wherein the control unit controls a camera path as the camera work. On the computer,
A procedure for calculating a distance distribution of the output image for each unit region predetermined in the output image based on the input image;
A procedure for controlling camera work in the output image as a moving image according to the distance distribution;
Camera work control program for executing