JPWO2015087368A1 - Depth information editing apparatus, depth information editing method, depth information editing program, and computer-readable recording medium recording depth information editing program - Google Patents

Abstract

An object of the present invention is to provide a program, an apparatus, a method, and the like that facilitate the editing and generation of depth information associated with an omnidirectional image. First, a means for inputting a drawing depth value, a means for drawing the drawing depth value in the depth map of the drawing image based on the coordinates, and a drawing image based on the line-of-sight information before projecting the drawing image onto the omnidirectional image And means for converting the depth map into a distance map. Second, after projecting an omnidirectional image onto a screen image, means for converting a distance map of the screen image into a depth map based on the line-of-sight information, and means for extracting a drawing depth value from the depth map of the screen image based on the coordinates And have.

Description

  The present invention provides, for example, a depth information editing apparatus, a depth information editing method, a depth information editing program, and the like, which edit and generate depth information associated with a so-called omnidirectional image, for example, an image that looks like a background covering the omnidirectional of the creator The present invention relates to a computer-readable recording medium on which a depth information editing program is recorded.

  Conventionally, a depth map is used as information representing the depth with respect to the position of each pixel of an image. Here, the “depth map” generally represents distance information from the camera (viewpoint) in a three-dimensional space in gray scale or the like.

  In other words, the depth map can be said to be a set of information representing the depth from the viewpoint to the target point (that is, a one-dimensional length with the perpendicular line from the viewpoint to the viewing frustum as an axis).

  In the field of computer graphics, the depth map is also called a Z buffer, and is used for determining the context of a three-dimensional object. The Z buffer can be said to be a memory area that stores the entire depth of the screen together.

  In recent years, depth maps have been used for expressing stereoscopic effects, depth of field, and the like.

  By the way, in the field of computer graphics, usually, three-dimensional information is generated first, so that a two-dimensional image and a depth map can be generated based on the three-dimensional information.

  On the other hand, in the field of drawing, drawing is usually performed on a two-dimensional image, and since three-dimensional information is not generated first, it is necessary to prepare a depth map separately.

International Publication No. WO / 2012/147303

  However, in the field of painting, a method for easily and accurately creating depth information has not been well established, and the creation takes an excessive amount of time.

  For example, when expressing a continuous change related to the depth of the ground or a building, an expression using gradation is performed based on the depth information. However, it is difficult to accurately express this by hand drawing.

  Furthermore, information limited to one direction such as a depth map is not suitable as depth information associated with an omnidirectional image. For example, in a depth map, in a direction perpendicular to a reference axis (such as a visual axis) The depth is always 0.

  Therefore, in the omnidirectional image editing apparatus described in Patent Document 1, which is a prior application by the present applicant, the method of projecting (drawing) from the drawing image to the omnidirectional image is directly used as a means for editing depth information. It is difficult to use.

  The present invention has been made in view of the above-described technical problems, and an object thereof is to realize editing and generation of depth information associated with an omnidirectional image.

  The depth information editing apparatus according to the first aspect of the present invention includes: means for inputting a drawing depth value; means for drawing the drawing depth value in a depth map of a drawing image based on coordinates; Means for converting the depth map of the drawing image into a distance map.

  The depth information editing apparatus according to the second aspect of the present invention includes means for inputting a drawing depth value, means for drawing the drawing depth value in a depth map of a drawing image based on coordinates, and the drawing image in all directions. Means for converting a depth value into a distance value at a timing of updating pixels of an omnidirectional image when projecting onto an image.

  The depth information editing apparatus according to the third aspect of the present invention is a means for converting a distance map of a screen image into a depth map based on line-of-sight information, and extracts a drawing depth value from the depth map of the screen image based on coordinates. And means for carrying out the above.

  The depth information editing apparatus according to the fourth aspect of the present invention, when projecting an omnidirectional image on a screen image, converts a distance value into a depth value at a timing of updating the image of the screen image, and coordinates based on the coordinates. And a means for extracting a drawing depth value from a depth map including the depth value of the screen image.

  A depth information editing apparatus according to a fifth aspect of the present invention includes an input unit that performs an operation input, a central control unit that performs a predetermined operation based on a depth information editing program, and a display unit that displays an image. The central control means draws a depth map on the screen image based on the distance map of the omnidirectional image, receives coordinates, buttons and numerical values from the input means, and when a line-of-sight change input is made, When the line-of-sight direction, viewing angle, and shift amount are updated based on the input information and depth value extraction is input, the screen image pixels are extracted based on the coordinates based on the input information and set as the drawing depth value. When a depth value change input is made, the setting value based on the input information is set as the drawing depth value. When a drawing input is made, the coordinates and the draw based on the input information are set. Update the pixel drawing image using queuing depth values and the drawing parameters, superimposed on the screen image depth map drawing image as a preview, and displaying on the display means.

  In another aspect of the present invention, the problem is solved as a computer-readable recording medium recording a depth information editing program, a depth information editing method, and a depth information editing program having the same meanings as the first to fifth aspects.

  According to the present invention, a depth information editing apparatus, a depth information editing method, a depth information editing program, and a depth information editing program that make it easy to edit and generate depth information associated with an omnidirectional image. A medium can be provided.

It is a block diagram of the depth information editing apparatus which concerns on one Embodiment of this invention. It is a memory map of the main memory of the depth information editing apparatus which concerns on one Embodiment of this invention. It is a conceptual diagram which shows the data input / output relationship of the depth information editing program which concerns on one Embodiment of this invention. It is a flowchart which shows operation | movement of the depth information editing program which concerns on one Embodiment of this invention. It is a flowchart which shows the process of FIG.4 S13 in detail. It is a flowchart which shows the pixel update process of an omnidirectional image. It is a flowchart which shows the process of FIG.4 S1 in detail. It is a flowchart which shows the pixel update process of a screen image. (A) thru | or (c) is a conceptual diagram which shows the relationship between a depth map and a distance map. It is a conceptual diagram for demonstrating the mutual conversion of a depth map and a distance map. (A) And (b) is a figure for demonstrating an example of the work procedure according to the depth information editing method which concerns on one Embodiment of this invention.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, exemplary embodiments of a depth information editing apparatus, a depth information editing method, a depth information editing program, and a computer-readable recording medium on which a depth information editing program is recorded will be described with reference to the drawings. It should be noted that the depth information editing program of the present invention is not limited to the following description, and can be appropriately changed without departing from the gist of the present invention.

  The depth information editing apparatus or the like of the present invention has at least the following as a part of its features.

  That is, first, the depth information editing apparatus according to the embodiment of the present invention includes a means for inputting a drawing depth value, a means for drawing the drawing depth value in the depth map of the drawing image based on the coordinates, and a drawing image. Means for converting a depth map of a drawing image into a distance map based on line-of-sight information before projecting onto an omnidirectional image. Here, the line-of-sight information includes, for example, information on the viewing angle and the shift amount. However, it is not limited to this.

  Second, the depth information editing apparatus according to the embodiment of the present invention includes a unit that converts a distance map of a screen image into a depth map based on line-of-sight information after projecting an omnidirectional image onto the screen image, and a coordinate based on the coordinates. Means for extracting a drawing depth value from a depth map of the screen image. Here, the line-of-sight information includes, for example, information on the viewing angle and the shift amount. However, it is not limited to this.

  As a first effect of the present invention, depth information representing a three-dimensional plane or a three-dimensional curved surface can be easily and accurately edited in an omnidirectional image. For example, when the drawing is performed with the line-of-sight direction directed directly below, depth information as a three-dimensional plane representing the ground can be created. When the line-of-sight direction is made horizontal from there, an accurate gradation is generated from the depth information as an expression of a continuous depth sensation on the ground.

  As a second effect of the present invention, new depth information related to edited depth information can be easily and accurately edited. For example, when an object is extracted from an arbitrary position on the ground with the line of sight oriented horizontally, the depth value is used, and a drawing that extends upward from the depth value is used, so that an object with a high height (for example, a building) Can be created.

  Hereinafter, an embodiment of the present invention will be described in detail.

  FIG. 1 shows and describes the configuration of a depth information editing apparatus according to an embodiment of the present invention.

  As shown in FIG. 1, the depth information editing apparatus 1 includes a personal computer 10, a display device 20, a pointing device 30, and the like.

  The depth information editing apparatus 1 corresponds to an aspect of a computer program product in which a depth information editing program that implements functions to be described later is installed.

  The personal computer 10 includes a central control unit 11 including a multitasking processor, a main memory 12 including a RAM (Random Access Memory) as a temporary storage device, an image control unit 13 such as a graphic card, and an input / output control unit 14. A built-in nonvolatile storage medium 15 and a media read / write interface 16.

  The image control unit 13 also includes a video memory 13a. Similar to the main memory 12 of the personal computer 10 main body, the video memory 13a is a place for temporarily storing data, and the memory attached to the graphic card is also referred to as VRAM (Video RAM). When a 3D graphic image is displayed on the screen, the amount of data required at that time increases. The data that has been processed by the image controller 13 is stored in the video memory 13a and used as needed. As the capacity of the video memory 13a is increased, even a fine 3D graphic image can be displayed smoothly and without a defect. In recent years, the speed of VRAM has been increased, and a memory standard dedicated to high-speed processing called GDDR (Graphics DDR) has appeared, and high-speed transfer of enormous data in three-dimensional graphics drawing has been realized.

  The display device 20 is a device that can display an image, such as a liquid crystal display. The pointing device 30 is a device capable of coordinate input and / or button input represented by a mouse, a touch panel, and a pen tablet.

  Program data 50, orientation pixel association data 42, and omnidirectional image input data 40 are input via the media read / write interface 16, and omnidirectional image output data 41 is output via the media read / write interface 16.

  The program data 50 is software that enables the depth information editing apparatus of the present invention to operate. This corresponds to data of a depth information editing program, which will be described later. The program data 50 may be recorded on a computer-readable recording medium and read from the recording medium.

  The azimuth image association data 42 is a table or function that associates azimuths and pixel positions with each other. When a polygon model is used, the polygon model data corresponds to this and becomes a three-dimensional object. The azimuth image association data 42 may accompany the program data 50, or may read externally defined data.

  The omnidirectional image input data 40 and the omnidirectional image output data 41 are image groups handled by software. For example, when a polygon model is used, a texture image group corresponds to this. The input texture image group and the like are temporarily stored in the main memory 12.

  The omnidirectional image input data 40 and the program data 50 may be read from an external recording medium (not shown) that can be read by the computer 10, received from an external computer via a communication network, and imported to the built-in nonvolatile storage medium 15. Good. Further, the output data may be written on an external recording medium (not shown) or transmitted to an external computer via a communication network.

  Here, FIG. 2 shows an example of a memory map of the main memory 12 used when editing depth information using a polygon model.

  The omnidirectional image related information is information for representing a background (that is, an omnidirectional image) covering the omnidirectional directions of the creator and the viewer in the depth information editing apparatus. FIG. 2 shows an example of using a polygon model. The polygon vertex group has three-dimensional space coordinates (x: horizontal, y: vertical, z: depth) and two-dimensional texture coordinates (u: horizontal, v: vertical). Since the polygon surface represents a triangle, the polygon surface group has three references to the polygon vertices and references to the texture image for the number of layers. Since the polygon model represents a three-dimensional shape, it has polygon vertices and polygon surfaces as arrays. When the structure of the texture image group of one layer is the same for every layer, the reference to the texture image may be one relative reference in the layer.

  The use of the polygon model is merely an example and does not technically limit the means for displaying and editing the omnidirectional image. If the pixels are associated in all directions, it is not essential to use the concept of polygon model or texture. For example, it is possible to edit an omnidirectional image obtained from an omnidirectional camera using a spherical mirror.

  Each of the texture image, the screen image, and the drawing image has a color with opacity (a: alpha, r: red, g: green, b: blue) as a two-dimensional array (hereinafter referred to as a color information map and Called). “Alpha” indicates opacity. As an example, a personal computer records the color of one pixel in units of 24 bits (8 bits per color, 8 × 3 = 24 bits for three colors of red, green, and blue). Values such as red, green, and blue are also called “density values”, and 256-bit recording is possible with the 8-bit type. In the case of a gray scale image, it is sufficient to have one density value.

  In the PNG with alpha (32-bit PNG: 32-bit Portable Network Graphics) format, in addition to color information, the opacity of each pixel can be recorded in 256 steps of 8-bit type. An alpha value of zero means completely transparent, 255 means completely opaque. As an example of an application for accurately handling colors in image processing, a format in which density values are expanded to 65536 levels of 16-bit type, a format represented by a floating-point type, or the like may be used.

  Furthermore, in the embodiment of the present invention, each of the texture image, the screen image, and the drawing image has a depth (a: alpha, g: density value) with opacity as a two-dimensional array. In the depth information editing apparatus and the like according to the embodiment of the present invention, two types of depth information, “depth map” and “distance map”, are handled.

  Here, the “depth map” is a value representing the depth from the viewpoint to the target point (that is, a one-dimensional length about a perpendicular line from the viewpoint to the viewing frustum) (hereinafter referred to as a depth value). Is a set of The “distance map” is a set of values (hereinafter referred to as distance values) representing a linear distance (that is, a three-dimensional length) between the viewpoint and the target point. In the depth map and the distance map, as in the color information map, it is assumed that the corresponding pixel can be designated by coordinates.

  Note that the depth value and the distance value may be given to the map as density values, instead of the density values proportional to the length, depending on the operation mode. For example, when it is desired to handle a length corresponding to infinity but the accuracy may decrease as the distance increases, it is more appropriate to use the reciprocal of the length.

  A screen image is an image which projects an omnidirectional image on a two-dimensional coordinate plane by coordinate transformation and presents it to the operator by the display device 20.

  Drawing is an operation of drawing a figure or a line on a two-dimensional (planar) image mainly using the pointing device 30. An image for drawing (drawing image) is a two-dimensional (planar) image that is a target on which the creator actually draws.

  The drawing image is in a different storage area from the screen image, has the same coordinate system as the screen image, and is completely transparent before drawing starts. The density value including opacity is updated at the location where the drawing has been performed. This is equivalent to the operator drawing on a transparent layer overlying the screen image.

  The drawing parameter is a parameter for determining a drawing technique and attributes. Examples of the drawing method include freehand, straight line, rectangle, circle, image pasting, and the like. The drawing attribute may be a width or density value for a line, a filled pattern for a figure, and the like. The more the types of drawing parameters are, the more expression methods can be used. Note that “drawing” in the present application conceptually includes general “painting”.

  In the present embodiment, the depth information of the texture image (that is, the omnidirectional image) is input / output as a distance map. The depth information of the screen image is input (projected) from the omnidirectional image as a distance map, and output (displayed) to the display device 20 as a depth map with conversion described later. The depth information of the drawing image is input (drawn) from an input device such as the pointing device 30 as a depth map, and is output (projected) as a distance map to an omnidirectional image with conversion described later. However, when outputting (displaying) together with the screen image to the display device 20, it is combined with the screen image as a depth map.

  The drawing image is also a temporary image until the contents of the drawing are reflected in the omnidirectional image, and may be used as necessary. For example, if it is not necessary to return to the line-of-sight change mode (correct the line-of-sight information) again before drawing, and the drawing rule is to immediately convert the drawing content and reflect it in the omnidirectional image. It is also possible not to use an image.

  FIG. 3 conceptually shows and explains the data input / output relationship of the depth information editing program according to an embodiment of the present invention.

  In the figure, an input device is a pointing device 30 or the like that receives an operation from an operator. The line-of-sight information means information for the operator to observe and edit an omnidirectional image, such as the viewing direction (yaw angle, pitch angle, roll angle), viewing angle, horizontal shift amount, and vertical shift amount. Yes. The input projection and the output projection mean input / output related to an omnidirectional image.

  The depth / distance conversion means that the depth map of the drawing image is converted into a distance map based on the line-of-sight information before the drawing image is projected onto the omnidirectional image. Distance / depth conversion means that a distance map of a screen image is converted into a depth map after an omnidirectional image is projected onto the screen image.

  A depth map is not suitable as depth information for an omnidirectional image because it is limited to a specific direction, but has a feature that drawing is intuitive and easy. For example, when it is desired to represent a three-dimensional plane, the area may be filled with a certain depth value. On the other hand, the distance map is not suitable for intuitive drawing, but has a feature that is not limited to a specific direction. That is, it can be used as depth information of an omnidirectional image. The depth map and the distance map can be converted into each other. The conversion will be described later.

  Similar to the color information map, the distance map is information including density values, so that projection onto an omnidirectional image and projection from an omnidirectional image are possible. It may also be affected by opacity when projection or composition is performed. That is, by controlling the opacity at the time of drawing, it is possible to prevent unnecessary areas from being drawn and to express the curved surface of the depth.

  The operation based on the depth information editing program according to the embodiment of the present invention will be described below with reference to the flowchart of FIG. At least a part of this series of operations corresponds to a depth information editing method according to an embodiment of the present invention.

  This processing is generally common to the processing of the conventional omnidirectional image editing program, but what corresponds to the conventional color information is depth information, and what corresponds to the conventional color information map is the depth map and distance map. It is. Further, a process related to projection (that is, a process of drawing a “distance map” based on the “depth map” of the drawing image on the omnidirectional image, and a “depth map” based on the “distance map” of the omnidirectional image. Is converted to a screen image), the conversion described later with reference to FIGS. Although not shown, not only the depth map but also the color information map may be edited by the same operation using a conventional method.

  In the depth information editing apparatus, when this depth information editing program is read from a recording medium or the like, loaded in the main memory 12, and executed by the central control unit 11, the central control unit 11 firstly selects the “distance” of the omnidirectional image. A “depth map” based on the “map” is drawn on the screen image (step S1), and coordinates, buttons, and numerical values are input from an input device such as the pointing device 30 (step S2).

  Subsequently, the central control unit 11 determines whether or not a line-of-sight change input has been made via an input device such as the pointing device 30 (step S3). If a line-of-sight change input has been made (YES in step S3). (Branch), the line-of-sight direction, viewing angle, and shift amount are updated based on the input information (step S4), and the process returns to step S1.

  When the line-of-sight change input has not been made (step S3 branches to NO), the central control unit 11 determines whether or not the depth value extraction input has been made (step S5), and when the depth value extraction input has been made. (Step S5 branches to YES), based on the coordinates of the input information, extract the pixel (depth value) of the screen image (depth map) and set it as the “drawing depth value” (step S6), the next step Proceed to S7. On the other hand, if the depth value extraction input has not been made (step S5 is branched to NO), the process proceeds to the next step S7 as it is.

  Subsequently, the central control unit 11 determines whether or not a depth value change input has been made via an input device such as the pointing device 30 (step S7). If a depth value change input has been made (step S7). (Branch to YES), the set value by the input information is set as the “drawing depth value” (step S8), and the process proceeds to the next step S9. On the other hand, when the depth value change input has not been made (step S7 is branched to NO), the process directly proceeds to the next step S9.

  Next, the central control unit 11 determines whether or not a drawing input has been made (step S9). If a drawing input has been made (step S9 branches to YES), the coordinates based on the input information, “drawing depth value” And the drawing image pixels are updated using the drawing parameters (step S10), the “depth map” of the drawing image is superimposed on the screen image as a preview and displayed on the display device 20 (step S11), and the next step S12 is performed. move on. On the other hand, if no drawing input has been made, the process proceeds directly to the next step S12.

  Subsequently, the central control unit 11 determines whether or not to end the drawing (step S12). When the drawing ends (step S12 branches to YES), based on the “depth map” of the drawing image. The “distance map” is drawn on the omnidirectional image (step S13), and the process returns to step S1. On the other hand, if the drawing is not finished, the process returns to step S2 and the above operation is repeated.

  As a condition for determining that the drawing is finished, it is assumed that the pointing device 30 is a mouse, and the timing at which the mouse button is released during dragging may be used. For example, a button for executing a drawing end process is not limited thereto. Or the like may be prepared, and the drawing may be ended when the button is pressed.

  Although not shown, the program can be terminated by a general method for terminating the program (for example, assuming that the pointing device 30 is a mouse and the program termination button is clicked). You may make it complete | finish a process.

  Next, the process executed in step S13 of FIG. 4 will be described in more detail with reference to the flowchart of FIG.

  In the drawing image, the depth information is handled by the depth map so that the creator can intuitively edit the drawing image. However, in consideration of handling the depth information by the distance map so that the omnidirectional image does not depend on the line of sight, step S13. Then, the depth map is converted into a distance map.

  Since the conversion can be easily performed in the screen coordinate system, the conversion timing is set before the drawing image is projected onto the omnidirectional image.

  That is, in the process of drawing the “distance map” based on the “depth map” of the drawing image on the omnidirectional image, in detail, the “depth map” of the drawing image is converted to the “distance map” using the viewing angle and the shift amount. ”(Step S21), the“ distance map ”of the drawing image is drawn on the omnidirectional image using the line-of-sight direction, the viewing angle, and the shift amount (step S22), and the drawing image is cleared (step S23). Return to the process of 4.

  However, the process of converting the depth map of the drawing image to the distance map of the omnidirectional image based on the line-of-sight information is not limited to the process executed in FIG. For example, when projecting a drawing image onto an omnidirectional image, the “depth value” may be converted into a “distance value” at the timing of updating the pixels of the omnidirectional image. In this case, the conversion timing may be after the pixels of the omnidirectional image are calculated from the pixels of the drawing image (that is, corresponding to a part of the process of projecting the drawing image onto the omnidirectional image). This is different in calculation method from the above-described conversion timing before projecting the drawing image onto the omnidirectional image, but can improve the accuracy of the interpolation (interpolation) processing.

  In this case, for the omnidirectional image pixel update process, as shown in FIG. 6, when the omnidirectional image pixel update process starts, a three-dimensional vector corresponding to the pixel of the omnidirectional image to be updated is calculated (step S31) Extracting the opacity and the depth value from the drawing image pixel corresponding to the azimuth of the three-dimensional vector and the line-of-sight information, and the neighboring pixels, respectively (step S32), from the opacity and depth value of each pixel One opacity and depth value are interpolated (step S33), a distance value is calculated from the interpolated depth value using the viewing angle and the shift amount (step S34), and the calculated opacity and distance value are used. Then, the pixels of the omnidirectional image are updated (step S35), and the process returns to the process of FIG.

  Next, the process executed in step S1 of FIG. 4 will be described in more detail with reference to the flowchart of FIG.

  While the depth information is handled by the distance map so that the omnidirectional image does not depend on the line of sight, the screen image is processed by the depth map so that the creator can intuitively edit the depth information. Then, a distance map is converted into a depth map.

  Since the conversion can be easily performed in the screen coordinate system, the conversion timing is after the omnidirectional image is projected onto the screen image.

  That is, in the process of drawing the “depth map” on the screen image based on the “distance map” of the omnidirectional image, in detail, the “distance map” of the omnidirectional image is used by using the viewing direction, the viewing angle, and the shift amount. Is drawn on the screen image (step S40), the “distance map” of the screen image is converted into the “depth map” using the viewing angle and the shift amount (step S41), and the process returns to the process of FIG.

  However, the process of converting the omnidirectional image distance map to the screen image depth map based on the line-of-sight information is not limited to the process executed in FIG. For example, when projecting an omnidirectional image onto a screen image, the “distance value” may be converted into a “depth value” at the timing of updating the pixels of the screen image. In this case, the conversion timing may be before the pixels of the screen image are calculated from the pixels of the omnidirectional image (that is, corresponding to a part of the process of projecting the omnidirectional image onto the screen image). This is different in calculation method from the above-described conversion timing after the omnidirectional image is projected onto the screen image, but the accuracy of the interpolation (interpolation) process can be improved.

  In this case, as shown in FIG. 8, in the pixel update process of the screen image, when the pixel update process of the screen image starts, a three-dimensional vector corresponding to the pixel of the screen image to be updated and the line-of-sight information is calculated ( (Step S51) The opacity and distance value are extracted from the pixels of the omnidirectional image corresponding to the orientation of the three-dimensional vector and the neighboring pixels (Step S52), and the viewing angle and the shift amount are used for each pixel. A depth value is calculated from the distance value (step S53), one opacity and depth value is interpolated from the opacity and depth values of each pixel (step S54), and the screen is obtained using the interpolated opacity and depth values. The pixel of the image is updated (step S55), and the process returns to the process of FIG.

  FIGS. 9A to 9C show the relationship between the depth map and the distance map.

  That is, FIGS. 9A to 9C both show the viewpoint and the image in the three-dimensional space viewed from the side. In order to illustrate the concept in an easy-to-understand manner, each map is represented by a one-dimensional array.

  In FIG. 9A, the horizontal axis represents the depth value, the vertical axis represents the vertical direction of the image, and the depth value of each pixel in the depth map is represented by a bold line. Here, the depth value of each pixel is constant.

  FIG. 9B shows the depth value on the horizontal axis and the vertical direction of the image on the vertical axis, and represents the depth value in the graph of FIG. 9A. The length of the thick line represents the optical axis in the line-of-sight direction. Represents the distance value of each pixel in an example in which is orthogonal at the center of the image.

  FIG. 9C shows a case where the horizontal axis is the distance value, the vertical axis is the vertical direction of the image, and the depth map in the graph of FIG. 9A is converted into a distance map.

  When the pixel position is at the center of the image, the depth value matches the distance value, and when the pixel position is far from the image center, the ratio of the distance value to the depth value increases.

  Here, with reference to FIGS. 10A and 10B, the mutual conversion between the depth map and the distance map will be described in detail. FIG. 10A shows a viewpoint and an image in a three-dimensional space viewed from the side, and FIG. 10B shows a viewpoint and an image in a three-dimensional space viewed from the front.

More specifically, the distance value distanceVal (i, j) corresponding to the predetermined depth value depthVal with respect to the position (i, j) of the image having the width width and height height when the viewing angle fov is It can be obtained by such a calculation.
depthBase = (height / 2) / tan (fov / 2)
distanceBase (i, j) = sqrt ((i-width / 2) ^ 2 + (j-height / 2) ^ 2 + depthBase ^ 2)
distanceCor (i, j) = distanceBase (i, j) / depthBase
distanceVal (i, j) = depthVal * distanceCor (i, j)

Thereby, the conversion performed on the drawing image, that is, the conversion from the depth map depthMap (i, j) to the distance map distanceMap (i, j) is obtained for each pixel as follows.
distanceMap (i, j) = depthMap (i, j) * distanceCor (i, j)

On the contrary, the conversion performed on the screen image, that is, the conversion from the distance map distanceMap (i, j) to the depth map depthMap (i, j) is obtained for each pixel as follows.
depthMap (i, j) = distanceMap (i, j) / distanceCor (i, j)

However, in the case of operation in which the density value of the distance map is the reciprocal of the length, each conversion may be obtained by switching multiplication and division as follows.
distanceMap (i, j) = depthMap (i, j) / distanceCor (i, j)
depthMap (i, j) = distanceMap (i, j) * distanceCor (i, j)

The above-described calculation of distanceBase (i, j) shows an example in which the optical axes are orthogonal at the center of the image. When the optical axis is not the center of the image, that is, when the horizontal shift amount shiftX and the vertical shift amount shiftY are handled, the following may be obtained.
distanceBase (i, j) = sqrt ((i-width / 2-shiftX) ^ 2 + (j-height / 2-shiftY) ^ 2 + depthBase ^ 2)

  Next, FIGS. 11A and 11B show an example of a procedure for drawing depth information on the ground and a building in accordance with the depth information editing program of the present invention. FIG. 11A shows the state of drawing, and FIG. 11B shows the state of confirmation.

  As shown in FIG. 11A, in the state of drawing, the depth map is represented in gray scale before and after drawing. Here, the density value of the depth map is the reciprocal of the length, and is expressed so as to approach white as the density value decreases (that is, the depth value increases and increases).

  However, the depth map is not limited to the gray scale expression, but may be an isoline, a hue, or the like. For example, the depth map and the color information map may be combined and displayed by expressing the depth map by hue and the color information map by brightness. In this case, the operator can easily specify the position to be noted on the color information map and the position on the corresponding depth map.

  In the example of FIG. 11 (a), the painter drew a circle facing down as an expression of the ground, and drawn one rectangle facing the front and one rectangle facing the front left as an expression of the building. Shall. All figures are filled with a certain depth value. An arbitrary value is applied as the drawing depth value in the first drawing procedure (for example, the height of the viewpoint with respect to the ground is assumed to be 1.00). As the drawing depth values in the second and third drawing procedures, depth values extracted from arbitrary positions of the screen image are applied.

  As shown in FIG. 11 (b), the confirmation shows that depth information regarding the ground and the building changes continuously as a three-dimensional plane, and that these are generated (displayed) with an accurate gradation. Yes.

  As described above in detail, according to an embodiment of the present invention, depth information representing a three-dimensional plane or a three-dimensional curved surface can be easily and accurately edited in an omnidirectional image. Furthermore, according to an embodiment of the present invention, new depth information related to edited depth information can be edited easily and accurately.

DESCRIPTION OF SYMBOLS 1 Depth information editing apparatus 10 Personal computer 11 Central control part 12 Main memory 13 Image control part 14 Input / output control part 15 Built-in non-volatile storage medium 16 Media read / write interface 20 Display apparatus 30 Pointing apparatus 40 Omnidirectional image input data 41 Omnidirectional image Output data 42 Orientation pixel association data 50 Program data

Claims (22)

Means for entering a drawing depth value;
Means for drawing the drawing depth value in the depth map of the drawing image based on the coordinates;
A depth information editing apparatus comprising: means for converting the depth map of the drawing image into a distance map based on line-of-sight information. The depth information editing apparatus according to claim 1, wherein the depth map of the drawing image is converted into a distance map based on the line-of-sight information before the drawing image is projected onto an omnidirectional image. Means for entering a drawing depth value;
Means for drawing the drawing depth value in the depth map of the drawing image based on the coordinates;
A depth information editing device comprising: a unit that converts a depth value into a distance value at a timing at which pixels of the omnidirectional image are updated when the drawing image is projected onto the omnidirectional image. The depth information editing apparatus according to claim 3, wherein the depth value is converted into a distance value after the pixels of the omnidirectional image are calculated from the pixels of the drawing image. Means for converting a distance map of a screen image into a depth map based on line-of-sight information;
A depth information editing device comprising: means for extracting a drawing depth value from the depth map of the screen image based on coordinates. The depth information editing apparatus according to claim 5, wherein the distance map of the screen image is converted into the depth map based on the line-of-sight information after the omnidirectional image is projected onto the screen image. Means for converting a distance value into a depth value at a timing of updating the image of the screen image when projecting an omnidirectional image on the screen image;
Means for extracting a drawing depth value from a depth map including the depth value of the screen image based on coordinates; and a depth information editing apparatus. The depth information editing apparatus according to claim 7, wherein the timing for converting the distance value into the depth value is before the pixels of the screen image are calculated from the pixels of the omnidirectional image. Input means for performing operation input;
Central control means for performing a predetermined operation based on the depth information editing program;
Display means for displaying an image,
The central control means includes
Draw a depth map on the screen image based on the distance map of the omnidirectional image,
Receiving at least one of coordinates, buttons and numerical values from the input means;
When the line-of-sight change input is made, the line-of-sight information is updated based on the input information. When the depth value extraction input is made, the screen image pixels are extracted based on the coordinates based on the input information, and the drawing is performed. When the depth value is changed and the depth value change input is made, the setting value by the input information is set as the drawing depth value,
When a drawing input is made, the coordinates of the input information, the drawing depth value and the drawing parameter are used to update the pixel of the drawing image, the depth map of the drawing image is superimposed on the screen image as a preview, and the display means Depth information editing device to be displayed. The depth information editing apparatus according to claim 9, wherein the central control unit draws a distance map based on a depth map of the drawing image in an omnidirectional image when the drawing ends. The depth information editing device according to any one of claims 1 to 10, wherein the line-of-sight information includes at least one of a line-of-sight direction, a viewing angle, and a shift amount. Computer
Means for entering a drawing depth value;
Means for drawing the drawing depth value in the depth map of the drawing image based on the coordinates;
A depth information editing program for functioning as means for converting the depth map of the drawing image into a distance map based on line-of-sight information before projecting the drawing image onto an omnidirectional image. Computer
Means for projecting an omnidirectional image onto a screen image and converting a distance map of the screen image into a depth map based on line-of-sight information;
A depth information editing program for functioning as means for extracting a drawing depth value from the depth map of the screen image based on coordinates. The depth information editing program according to claim 12 or 13, wherein the line-of-sight information includes at least one of a line-of-sight direction, a viewing angle, and a shift amount. Computer
Draw a depth map on the screen image based on the distance map of the omnidirectional image,
Receive at least one of coordinates, buttons and numerical values from the input device,
When a line-of-sight change input is made, the line-of-sight direction, the viewing angle and the shift amount are updated based on the input information. When a depth value extraction input is made, the pixel of the screen image is based on the coordinates based on the input information. Is extracted and set as the drawing depth value.When a depth value change input is made, a setting value based on input information is set as the drawing depth value.
When a drawing input is made, the pixel of the drawing image is updated using the coordinates based on the input information, the drawing depth value, and the drawing parameter, and the depth map of the drawing image is superimposed on the screen image as a preview and is displayed on the display device. Depth information editing program to display and function as a central control means. The depth information editing program according to claim 15, wherein the central control unit draws a distance map based on a depth map of the drawing image in an omnidirectional image when the drawing ends. Entering a drawing depth value;
Drawing the drawing depth value into the depth map of the drawing image based on the coordinates;
Converting the depth map of the drawing image into a distance map based on line-of-sight information before projecting the drawing image onto an omnidirectional image. Transforming the distance map of the screen image into a depth map based on the line-of-sight information after projecting the omnidirectional image onto the screen image;
Extracting a drawing depth value from the depth map of the screen image based on coordinates; and a depth information editing method. Computer
Means for entering the drawing depth value,
Means for drawing the drawing depth value into a depth map of a drawing image based on coordinates;
Before projecting the drawing image onto an omnidirectional image, a computer-readable recording that records a depth information editing program for functioning as means for converting the depth map of the drawing image into a distance map based on line-of-sight information Medium. Computer
Means for converting a distance map of a screen image into a depth map based on line-of-sight information after projecting an omnidirectional image onto the screen image;
A computer-readable recording medium recording a depth information editing program for functioning as a means for extracting a drawing depth value from the depth map of the screen image based on coordinates. Computer
Draw a depth map on the screen image based on the distance map of the omnidirectional image,
Receive at least one of coordinates, buttons and numerical values from the input device,
When a line-of-sight change input is made, the line-of-sight direction, the viewing angle and the shift amount are updated based on the input information. When a depth value extraction input is made, the pixel of the screen image is based on the coordinates based on the input information. Is extracted and set as the drawing depth value.When a depth value change input is made, a setting value based on input information is set as the drawing depth value.
When a drawing input is made, the pixel of the drawing image is updated using the coordinates based on the input information, the drawing depth value, and the drawing parameter, and the depth map of the drawing image is superimposed on the screen image as a preview and is displayed on the display device. A computer-readable recording medium on which a depth information editing program for displaying and functioning as central control means is recorded. The central control means, when finishing drawing, draws a distance map based on a depth map of the drawing image on an omnidirectional image, and the computer can read the depth information editing program according to claim 21. recoding media.