WO2024057902A1 - Dispositif, procédé et programme de traitement d'informations - Google Patents

Dispositif, procédé et programme de traitement d'informations Download PDF

Info

Publication number
WO2024057902A1
WO2024057902A1 PCT/JP2023/031088 JP2023031088W WO2024057902A1 WO 2024057902 A1 WO2024057902 A1 WO 2024057902A1 JP 2023031088 W JP2023031088 W JP 2023031088W WO 2024057902 A1 WO2024057902 A1 WO 2024057902A1
Authority
WO
WIPO (PCT)
Prior art keywords
depth map
depth
local area
local
local region
Prior art date
Application number
PCT/JP2023/031088
Other languages
English (en)
Japanese (ja)
Inventor
健吾 早坂
Original Assignee
ソニーグループ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーグループ株式会社 filed Critical ソニーグループ株式会社
Publication of WO2024057902A1 publication Critical patent/WO2024057902A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/62—Extraction of image or video features relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking

Definitions

  • the present disclosure relates to an information processing device, method, and program, and particularly relates to an information processing device, method, and program that allow editing of a local region in which the depth value of a depth map changes in the temporal direction.
  • the generated depth map does not always completely correspond to the image. For example, it is difficult to correctly derive the depth value of a subject that is difficult to detect using passive sensing or a subject that has no texture. Furthermore, there may be cases where an incorrect depth value is derived due to the influence of noise.
  • Patent Document 1 discloses that an assist depth map composed of depth data to be added (synthesized) to a basic depth map composed of depth values of a subject in a certain image is generated, and a synthetic depth map of the basic depth map and the assist depth map is created.
  • a method for generating is disclosed.
  • a correction value of the depth value is set for a local region to be corrected in the basic depth map.
  • the position of this local area can also be moved in the time direction. In that case, if a local area is set in the key frame, the local area in the frames between the key frames is interpolated.
  • the present disclosure has been made in view of this situation, and makes it possible to edit a local region in which the depth value of a depth map changes in the time direction.
  • An information processing device includes designation of a first local region in a first key frame of a basic depth map acquired by a depth sensor and corresponding to a moving image, and designation of a first local region corresponding to the first local region.
  • a reception unit that accepts designation of a local region depth map, designation of a second local region in a second key frame of the basic depth map, and designation of a second local region depth map corresponding to the second local region; , first designated depth data representing the first local region depth map, second designated depth data representing the second local region depth map, and the first key frame and the second key.
  • an assist depth map generation unit that generates an assist depth map including interpolated depth data representing an interpolated local area depth map corresponding to an interpolated local area depth map in at least one frame between frames, the first local area depth map; and the second local area depth map has different depth values
  • the interpolated local area is a local area that interpolates the first local area and the second local area
  • the interpolated local area depth map is a local area that interpolates the first local area and the second local area
  • the interpolated local area depth map is
  • the information processing apparatus is a depth map that interpolates the first local area depth map and the second local area depth map.
  • An information processing method includes specifying a first local region in a first key frame of a basic depth map acquired by a depth sensor and corresponding to a moving image, and specifying a first local region corresponding to the first local region.
  • This information processing method is a depth map that interpolates a second local region depth map.
  • a program causes a computer to specify a first local region in a first key frame of a basic depth map acquired by a depth sensor and corresponding to a moving image, and to specify a first local region corresponding to the first local region.
  • a reception desk that accepts designation of a local region depth map of , designation of a second local region in a second key frame of the basic depth map, and designation of a second local region depth map corresponding to the second local region. , first specified depth data representing the first local area depth map, second specified depth data representing the second local area depth map, and the first key frame and the second specified depth map.
  • the program functions as an assist depth map generation unit that generates an assist depth map including interpolated depth data representing an interpolated local area depth map corresponding to an interpolated local area in at least one frame between key frames,
  • the local area depth map and the second local area depth map have different depth values
  • the interpolated local area is a local area that interpolates the first local area and the second local area
  • the interpolated local area is a depth map that interpolates the first local region depth map and the second local region depth map.
  • An information processing device, method, and program include specifying a first local region in a first key frame of a basic depth map acquired by a depth sensor and corresponding to a moving image; designation of a first local region depth map corresponding to the basic depth map, designation of a second local region in a second key frame of the basic depth map, and a second local region depth map corresponding to the second local region. is accepted, and the first specified depth data representing the first local area depth map, the second specified depth data representing the second local area depth map, and the first key frame and the first specified depth data are received.
  • An assisted depth map is generated that includes interpolated depth data representing an interpolated local region depth map corresponding to an interpolated local region in at least one frame between two key frames.
  • the interpolation local area is a local area that interpolates the first local area and the second local area.
  • the interpolated local area depth map is a depth map that interpolates the first local area depth map and the second local area depth map.
  • FIG. 3 is a diagram for explaining an example in which a depth value changes in the time direction.
  • FIG. 3 is a diagram illustrating an example of a method for generating an assist depth map.
  • FIG. 3 is a diagram illustrating an example of a method for generating an assist depth map.
  • FIG. 3 is a diagram illustrating an example of a method for generating an assist depth map. It is a figure which shows the layer structure of a depth map, and an example of composition.
  • FIG. 6 is a diagram illustrating an example of interpolation of a dynamic local region depth map.
  • FIG. 6 is a diagram illustrating an example of linear interpolation of a dynamic local region depth map composed of a single depth value.
  • FIG. 3 is a diagram illustrating an example of a method for generating an assist depth map.
  • FIG. 6 is a diagram illustrating an example of nonlinear interpolation of a dynamic local region depth map composed of a single depth value.
  • FIG. 6 is a diagram illustrating an example of reference interpolation of a dynamic local area depth map composed of a single depth value.
  • FIG. 6 is a diagram illustrating an example of manual interpolation of a dynamic local region depth map composed of a single depth value.
  • FIG. 7 is a diagram illustrating an example of a local region depth map configured of a plurality of depth values, in which vertex depth values can be set. It is a figure which shows the example of a confirmation screen. It is a figure showing an example of a display.
  • FIG. 6 is a diagram illustrating an example of interpolation of a local region depth map whose shape is dynamic.
  • FIG. 3 is a diagram illustrating an example of how a local area is set.
  • FIG. 3 is a diagram illustrating an example of how a local area is set.
  • FIG. 3 is a diagram illustrating an example of how a local area is set.
  • 1 is a block diagram showing an example of the main configuration of a computer.
  • FIG. 2 is a functional block diagram showing functions implemented in a computer.
  • FIG. 3 is a diagram showing an example of a depth editing screen.
  • 3 is a flowchart illustrating an example of the flow of depth map generation processing.
  • 12 is a flowchart illustrating an example of the flow of local area depth map generation processing.
  • Patent Document 1 (mentioned above)
  • Assist depth map> ⁇ Edit depth map>
  • image processing techniques for generating depth maps from parallax images from two left and right viewpoints or from multiple viewpoints. For example, in the passive sensing algorithm framework, after calculating the matching cost, each person devised a way to leave highly reliable information, making corrections, and finally outputting a depth map. Additionally, a method has been proposed that uses a highly reliable depth map to determine the disparity value of a pixel with unknown depth.
  • the generated depth map does not always completely correspond to the image. For example, it is difficult to correctly derive the depth value of a subject that is difficult to detect using passive sensing or a subject that has no texture. Furthermore, there may be cases where an incorrect depth value is derived due to the influence of noise.
  • Patent Document 1 discloses that an assist depth map composed of depth data to be added (synthesized) to a basic depth map composed of depth values of a subject in a certain image is generated, and a synthetic depth map of the basic depth map and the assist depth map is created.
  • a method for generating is disclosed.
  • a correction value of the depth value is set for a local region to be corrected in the basic depth map.
  • the position of this local area can also be moved in the time direction. In this case, by setting the local area in the key frame, the local area can be placed at a position corresponding to the change in the time direction (movement of the local area between the key frames) in the frames between the key frames.
  • image 11 is the frame image of the t-th frame from the beginning (t frame) of the moving image
  • image 12 is the frame image of the frame X frames after that (t+X frame).
  • the automobile 21 has moved closer to the front in the image 12 than in the image 11.
  • transparent materials such as automobile windshields often transmit light or reflect light strongly, and it may be difficult to obtain the correct depth value.
  • the depth value of such a local region of the basic depth map can be corrected using the assist depth map.
  • the t frame and t+X frame of this moving image are used as key frames, and the local area 31 is set in the image 11 and the local area 32 is set in the image 12.
  • a local area that interpolates the local area 31 and the local area 32 is set in the frame between the t frame and the t+X frame.
  • the local area 32 of the image 12 has a depth value ( It is necessary to set a value smaller than the depth value of the local area 31.
  • the depth value of the local area to be interpolated cannot be set to a value that corresponds to the change in the depth value of the local area between key frames. Therefore, it is difficult to correct the depth value of the windshield portion of the automobile 21 in each frame from the t frame to the t+X frame to a correct value.
  • a depth map refers to a set of depth values for mutually different positions (more specifically, a group of depth values corresponding to a certain area). Further, in this specification, similarly to the method described in Patent Document 1, a depth map of a moving image is edited using an assist depth map (another depth map is combined with the basic depth map).
  • the basic depth map is a depth map consisting of the depth value of the subject of the moving image.
  • the basic depth map is composed of depth values acquired (detected) by a depth sensor. Therefore, the basic depth map does not necessarily accurately indicate the depth value of the subject (it may include a depth value different from the actual depth value of the subject).
  • the assist depth map is a depth map composed of depth data to be added (synthesized) to the basic depth map.
  • a composite depth map is generated by adding (combining) the depth data of the assist depth map to the basic depth map. That is, the basic depth map is edited using the assist depth map, and a composite depth map is obtained as the editing result.
  • the assist depth map may be composed of depth data to be added (synthesized) to a partial region of the basic depth map. That is, the basic depth map may be able to be edited for each partial region using the assist depth map.
  • the partial region in which the depth map (depth value) is edited in this way is also referred to as a local region.
  • the depth map of this local region included in the assist depth map is also referred to as a local region depth map.
  • changing in the time direction is also referred to as dynamic.
  • a local area depth map in which the depth value (or depth value distribution) changes in the time direction is also referred to as a dynamic local area depth map.
  • an overwritten depth map is also generated in addition to the basic depth map and the assist depth map.
  • the overwriting depth map is a depth map configured with depth data that updates the depth value of the basic depth map, and is a depth map for editing the basic depth map. That is, as shown on the left side of FIG. 5, an overwritten depth map 101, a basic depth map 102, and an assist depth map 103 are generated.
  • the overwritten depth map 101, the basic depth map 102, and the assist depth map 103 are generated as mutually different layers, and are combined (superimposed) by optimization processing.
  • the overwriting depth map 101 is given the highest priority during optimization processing.
  • the initial value of each pixel in the overwritten depth map 101 is set to an invalid value or the same value as the basic depth map 102.
  • the basic depth map 102 is given priority over the assist depth map 103 during optimization processing.
  • the assist depth map 103 is the layer with the lowest priority.
  • the depth value of the assist depth map 103 is adopted only when the depth value of that pixel in the overwriting depth map 101 and the basic depth map 102 is an invalid value.
  • the initial value of each pixel of the assist depth map 103 is set to an invalid value or the same value as the basic depth map 102.
  • each layer is combined, and the depth values of each layer are superimposed as seen from above in the figure, as shown by the arrow shown on the left side of FIG.
  • the overwrite depth map 101 is superimposed first
  • the basic depth map 102 is superimposed second to the front
  • the assist depth map 103 is superimposed last, resulting in the superimposition result shown on the right side of FIG.
  • a composite depth map 110 is generated. Therefore, the depth value of the assist depth map 103 acts only on areas that are invalid values in both the overwrite depth map 101 and the basic depth map 102.
  • the depth map may have a single layer structure.
  • method 1 described above is applied to generate a local area depth map (depth value or distribution of depth values) in the temporal direction. Make it possible to change.
  • the information processing device specifies a first local region in a first key frame of a basic depth map acquired by a depth sensor and corresponds to a moving image, and specifies a first local region depth map corresponding to the first local region. , a second local region in the second key frame of the basic depth map, and a second local region depth map corresponding to the second local region; First designated depth data representing a local region depth map, second designated depth data representing a second local region depth map, and at least one frame between the first key frame and the second key frame. and an assist depth map generation unit that generates an assist depth map including interpolated depth data representing an interpolated local area depth map corresponding to the interpolated local area.
  • the information processing method specifying a first local region in a first key frame of a basic depth map acquired by a depth sensor and corresponding to a moving image, and specifying a first local region depth map corresponding to the first local region. , the designation of the second local region in the second key frame of the basic depth map, and the designation of the second local region depth map corresponding to the second local region, and first specified depth data representing a map, second specified depth data representing a second local region depth map, and interpolated local region in at least one frame between the first key frame and the second key frame.
  • An assist depth map including interpolated depth data representing an interpolated local area depth map corresponding to is generated.
  • the program also causes the computer to specify a first local region in a first key frame of a basic depth map acquired by a depth sensor and corresponding to a moving image, and to specify a first local region depth corresponding to the first local region.
  • a reception unit that accepts designation of a map, designation of a second local region in a second key frame of the basic depth map, and designation of a second local region depth map corresponding to the second local region; first specified depth data representing a local region depth map of , second specified depth data representing a second local region depth map, and at least one frame between the first key frame and the second key frame. It functions as an assist depth map generation unit that generates an assist depth map including interpolated depth data representing an interpolated local area depth map corresponding to an interpolated local area in .
  • the interpolation local area is a local area that interpolates the first local area and the second local area.
  • the interpolated local area depth map is a depth map that interpolates the first local area depth map and the second local area depth map. That is, in this specification, a local area that is set in a frame between key frames and that interpolates a local area of a key frame is also referred to as an interpolated local area. Further, the depth map of the interpolated local area is also referred to as an interpolated local area depth map. This interpolated local area depth map can also be said to be a depth map that interpolates the local area depth map of the key frame.
  • the user sets frame X of the moving image as a key frame, sets a local region in the assist depth map 121 of that frame X, and sets the local region depth map 131.
  • the user sets frame X+30 as a key frame, sets a local area corresponding to the local area of frame X in the assist depth map 123 of that frame Set.
  • the user sets the local area depth map 132 to a different depth map from the local area depth map 131. That is, the local region depth map 131 and the local region depth map 132 have different depth values or depth value distributions.
  • the assist depth map generation unit sets an interpolated local area (interpolated local area depth map) in frames between these key frames. For example, the assist depth map generation unit sets the interpolated local region depth map 133 in the assist depth map 122 of frame X+15. This interpolated local area depth map 133 is set to a different depth map from the local area depth map 131 and the local area depth map 132. That is, the local region depth map 131, the local region depth map 132, and the interpolated local region depth map 133 have different depth values or depth value distributions.
  • the user can easily set a dynamic local area depth map without having to set local area depth maps for all frames.
  • the local region depth map may be configured with a single depth value, as shown in the second row from the top of the table in FIG. 2 (Method 1-1).
  • the first local area depth map, the second local area depth map, and the interpolated local area depth map described above may each be configured with a single depth value. That is, in each frame, one depth value may be set for a local area as a local area depth map.
  • the user sets frame X of a moving image as a key frame, sets a local area in the assist depth map 121 of that frame It is set as the depth map 131.
  • the local area depth map 131 is configured with a depth value of "10”.
  • the user sets frame X+30 as a key frame, sets a local area corresponding to the local area of frame X in the assist depth map 123 of that frame It is set as a local area depth map 132.
  • the local area depth map 132 is configured with a depth value of "30".
  • the assist depth map generation unit sets an interpolated local area (interpolated local area depth map) in frames between these key frames. For example, in the assist depth map 122 of frame X+15, the assist depth map generation unit sets an interpolated local area corresponding to the local area of frame do.
  • the interpolated local area depth map 133 is configured with a depth value of "20". That is, the local region depth map 131, the local region depth map 132, and the interpolated local region depth map 133 have different depth values.
  • the user can easily set a dynamic local area depth map without setting local area depth maps for all frames. That is, it is possible to easily edit a local region where the depth value of the depth map changes in the time direction.
  • the depth value may be managed for each local region in the assist depth map.
  • depth values may be managed for each local area.
  • Method 1-1-1 When this method 1-1 is applied, as shown in the third row from the top of the table in Figure 2, the depth value of the local area of the key frame is linearly interpolated to generate an interpolated local area depth map.
  • the assist depth map generation unit sets the depth value of the interpolated local area depth map by linearly interpolating the depth value of the first local area depth map and the depth value of the second local area depth map. Good too. That is, in this case, an interpolated local area depth map in which the depth value changes linearly (linearly) in the time direction is generated.
  • the user sets a local area depth map 131 configured with a depth value of "10" in the assist depth map 121 of frame X of a moving image, and sets the assist depth map 131 of frame
  • a local area depth map 132 configured with a depth value of "30" is set.
  • the assist depth map generation unit sets an interpolated local region depth map by linear interpolation in each frame between these key frames (frame X and frame X+30).
  • the assist depth map generation unit sets an interpolated local region depth map corresponding to the local region depth map 131 and the local region depth map 132 in each frame, and uses the depth value of the interpolated local region depth map as the local region depth map.
  • the depth values of the map 131 and local area depth map 132 are set to values for linear interpolation.
  • An interpolated local region depth map 133 is set, which is configured by "20" (a value intermediate between the depth value "10" and the depth value "30").
  • the user can easily set a local region depth map in which the depth value changes linearly in the time direction without having to set the local region depth map for all frames.
  • the user can easily create a local region depth map whose depth value changes linearly in the time direction without having to manually generate an assist depth map for every frame.
  • Assist depth maps can be generated. That is, it is possible to easily edit a local region where the depth value of the depth map changes linearly in the time direction.
  • an interpolated local area depth map is generated by non-linearly interpolating the depth value of the local area of the key frame.
  • the assist depth map generation unit sets the depth value of the interpolated local area depth map by nonlinearly interpolating the depth value of the first local area depth map and the depth value of the second local area depth map. Good too. That is, in this case, an interpolated local region depth map in which the depth value changes nonlinearly (curvilinearly) in the time direction is generated.
  • the user sets a local region depth map 131 configured with a depth value of "10" in the assist depth map 121 of frame X of a moving image, and sets the assist depth map 131 of frame
  • a local area depth map 132 configured with a depth value of "30" is set.
  • the assist depth map generation unit sets an interpolated local area depth map by nonlinear interpolation in each frame between these key frames (frame X and frame X+30). That is, the assist depth map generation unit sets an interpolated local region depth map corresponding to the local region depth map 131 and the local region depth map 132 in each frame, and uses the depth value of the interpolated local region depth map as the local region depth map.
  • the depth values of the map 131 and the local area depth map 132 are set to values for nonlinear interpolation. For example, in the assist depth map 122 of frame X+15, which is between frame X and frame 15'' is set.
  • the interpolated local area depth map of each frame is set so that the depth value of this local area changes nonlinearly (curvilinearly) in the time direction.
  • the depth value of this interpolated local region depth map can be set in any manner.
  • the depth value of the interpolated local region depth map of each frame may be set based on a nonlinear predetermined function. Further, for example, a user or the like may be allowed to set this function. Further, the depth value of the interpolated local region depth map of each frame may be set to follow the movement of the entire screen (change in depth value).
  • the user can easily set a local region depth map in which the depth value changes non-linearly in the time direction, without having to set the local region depth map for all frames.
  • the user can easily create a local region depth map whose depth value changes non-linearly in the time direction without having to manually generate an assist depth map for every frame.
  • Assist depth maps can be generated. That is, it is possible to easily edit a local region where the depth value of the depth map changes nonlinearly in the time direction.
  • Method 1-1 when method 1-1 is applied, as shown in the fifth row from the top of the table in Figure 2, a reference object is recognized in the RGB image, and according to the depth value of the reference object between key frames.
  • An interpolated local area depth map may also be generated using the method (method 1-1-3). In this specification, such an interpolation method is also referred to as reference interpolation.
  • the reception unit receives the designation of a reference object that refers to the depth value
  • the assist depth map generation unit generates the depth value of the first local area depth map, the depth value of the second local area depth map, and the interpolation
  • the depth value of the local region depth map may be set based on the depth value of the reference object of each frame. That is, in the case of this reference interpolation, a reference object whose depth value is referred to is specified in an RGB image (moving image). Then, in each frame of the assist depth map, the depth value of the local region depth map is set based on the depth value of the reference object in that frame. In other words, in the case of this reference interpolation, an interpolated local region depth map is generated in which the change in depth value in the time direction is similar to that of the reference object.
  • any object that appears in each frame of the RGB image can be set as this reference object.
  • an object that includes the local area may be used as the reference object, or an object that is located at a location different from the local area may be used as the reference object.
  • the depth value of the local area depth map (including the interpolated local area depth map) of each frame may be set based on the depth value of the reference object, and any method may be used.
  • the depth value of the local area depth map may be the same depth value as the reference object of the frame, or may be a value obtained by performing a predetermined calculation on the depth value of the reference object of the frame.
  • a user sets a local area (local area depth map 131) in the assist depth map 121 of frame X of a moving image, and sets a local area (local area depth map 131) in the assist depth map 123 of frame A corresponding local area (local area depth map 132) is set. Further, the user specifies an object that includes these local regions as a reference object 141 for referring to the depth value.
  • the assist depth map generation unit sets the depth value of the local area depth map 131 to the depth value "10" of the reference object 141 of that frame, and sets the depth value of the local area depth map 132 to the depth value of the reference object 141 of that frame. Set the depth value to "30".
  • the assist depth map generation unit sets an interpolated local region depth map by reference interpolation in each frame between these key frames (frame X and frame X+30).
  • the assist depth map generation unit sets an interpolated local area based on the position of the reference object 141 in that frame, and creates the interpolated local area depth map based on the depth value of the reference object 141 in that frame.
  • Set the depth value For example, in the assist depth map 122 of frame X+15, which is between frame X and frame
  • the depth value of the interpolated local area depth map 133 is set to 40, which is the same as the depth value of the reference object 141 of that frame. That is, the interpolated local area depth map of each frame is set so that the depth value of this local area changes in the same way as the depth value of the reference object 141 in frames X to frame X+30.
  • the user can easily set a local region depth map in which the change in depth value in the time direction is similar to that of the reference object, without the user having to set the local region depth map for all frames.
  • the user can easily create a local region depth whose depth value changes in the time direction are similar to that of the reference object, without having to manually generate an assisted depth map for every frame.
  • An assisted depth map with a map can be generated. That is, it is possible to easily edit a local region where the depth value of the depth map changes in the same way as the depth value of the reference object in the time direction.
  • Method 1-1 when method 1-1 is applied, as shown in the sixth row from the top of the table in Figure 2, the amount of change in depth value per frame is set in the key frame, and the interpolation localization is performed according to the setting.
  • a region depth map may also be generated (method 1-1-4).
  • such an interpolation method is also referred to as manual interpolation.
  • the reception unit accepts a specification of the amount of change in depth value per frame
  • the assist depth map generation unit generates an interpolated local area depth based on the depth value of the first local area depth map and the amount of change thereof.
  • You may also set a depth value for the map. That is, in this case, the amount of change per frame in the depth value of the local area depth map is specified in the key frame.
  • the depth value of the interpolated local area depth map is set based on the depth value of the local area depth map of the immediately preceding key frame and the set amount of change. That is, in this case, the depth value of the local region depth map of each frame changes according to the set amount of change.
  • the user sets a local area depth map 131 configured with a depth value of "10" in the assist depth map 121 of frame X of a moving image, and sets the assist depth map 131 of frame
  • a local area depth map 132 configured with a depth value of "30” is set.
  • the assist depth map 121 of frame X the user sets "+1" as the amount of change 151 per frame in the depth value of the local region depth map 131.
  • the assist depth map generation unit sets an interpolated local region depth map by manual interpolation in each frame between these key frames (frame X and frame X+30).
  • the assist depth map generation unit sets an interpolated local region depth map corresponding to the local region depth map 131 and the local region depth map 132 in each frame, and uses the depth value of the interpolated local region depth map as the local region depth map.
  • the depth value of the map 131 is set to a value changed according to the amount of change 151.
  • the assist depth map 122 of frame X+15 which is between frame X and frame
  • the user can easily set a local region depth map in which the depth value changes by a desired amount of change in the time direction, without having to set the local region depth map for all frames.
  • a local region depth map that allows the depth value to easily change by a desired amount of change in the time direction without the user having to manually generate an assist depth map for every frame. It is possible to generate an assisted depth map having the following. That is, it is possible to easily edit a local region where the depth value of the depth map changes by a desired amount of change in the time direction.
  • an interpolated local region depth map may be generated using a neural network (method 1-1 -5).
  • This neural network is a neural network generated by prior learning so as to output an interpolated local area depth map based on the information of the local area of the key frame.
  • AI Artificial Intelligence
  • the assist depth map generation unit may set the depth value of the interpolated local area depth map using a neural network that outputs the interpolated local area depth map based on the information of the local area of the key frame. That is, in this case, for example, by inputting the local area depth map set in the key frame to the neural network, an interpolated local area depth map of each frame between the key frames can be obtained. That is, in this case, the depth value of the local region depth map of each frame changes depending on the specifications of the neural network.
  • the user can easily set a local region depth map in which the depth value changes in the time direction without having to set the local region depth map for all frames.
  • the user can easily create an assisted depth map that has a local area depth map whose depth value changes in the time direction without having to manually generate an assisted depth map for every frame. can be generated. That is, it is possible to easily edit a local region where the depth value of the depth map changes in the time direction.
  • the local area depth map may be configured with a plurality of depth values, as shown in the 8th row from the top of the table in FIG. 2 (method 1-2).
  • the first local area depth map, the second local area depth map, and the interpolated local area depth map described above may each be configured with a plurality of depth values. That is, in each frame, a local region depth map having a plurality of depth values may be set for the local region. That is, in this case, the depth values of at least some regions are different from the depth values of other regions in the local region depth map. For example, the depth value may be different for each pixel.
  • this local region depth map 160 does not have a single density, but is composed of a plurality of depth values. In other words, the depth value can change depending on the position (pixel) within this local area.
  • the assist depth map generation unit When the user sets such a local region depth map in a key frame, the assist depth map generation unit generates an interpolated local region (local region depth map) that interpolates the local region (local region depth map) of the key frame in frames between the key frames. interpolated local area depth map).
  • the method of this interpolation is arbitrary, and may be the same as, for example, the method 1-1 (which may include methods 1-1-1 to 1-1-5) described above. That is, the local area depth map (interpolated local area depth map) set in each frame is composed of a plurality of depth values.
  • the user can easily set a dynamic local region depth map without having to set the local region depth map for every frame. can. That is, it is possible to easily edit a local region where the depth value of the depth map changes in the time direction.
  • the vertex depth value of the local area depth map is set for each key frame, and the vertex depth value is Other depth values of the local region depth map may be set based on this (method 1-2-1).
  • the depth value of the vertex of a local region is also referred to as the vertex depth value.
  • the reception unit accepts the designation of each vertex depth value of the first local region and the designation of each vertex depth value of the second local region
  • the assist depth map generation unit receives the designation of each vertex depth value of the first local region.
  • the first local area depth map may be set based on the vertex depth value of the local area
  • the second local area depth map may be set based on the specified second local area vertex depth value.
  • the local region depth map 160 in FIG. By specifying a depth value (vertex depth value) for each, depth values are set for other pixels within the local area based on each specified vertex depth value. That is, the local area depth map 160 is set.
  • the user can easily set the local area depth map just by specifying the vertex depth value of the local area (without specifying the depth values of all pixels in the local area).
  • each vertex depth value may be managed for each local region in the assist depth map.
  • each vertex depth value may be managed for each local area.
  • any method can be used to set the depth values of other pixels in the local area based on each specified vertex depth value.
  • the vertex depth values may be linearly interpolated to set other depth values (method 1-2-1-1).
  • the assist depth map generation unit may set the first local area depth map by linearly interpolating the vertex depth values of the first local area.
  • the assist depth map generation unit may set the second local area depth map by linearly interpolating the vertex depth values of the second local area. That is, in this case, a local area depth map in which the depth value changes linearly (linearly) in the spatial direction is generated.
  • the user specifies the depth value of the vertex 161 as "50”, the depth value of the vertex 162 as “10”, and the depth value of the vertex 163 as "50". Assume that the depth value of the vertex 164 is specified as "100”.
  • the assist depth map generation unit linearly interpolates these vertex depth values to set depth values of other pixels of the local area depth map 160. As a result, the local region depth map 160 is set to have a distribution of depth values such that the values gradually decrease from the upper right to the lower left, for example.
  • ⁇ Method 1-2-1-2> when method 1-2-1 is applied, other depth values may be set by non-linear interpolation of the vertex depth values, as shown at the bottom of the table in FIG. 2 (method 1-2-1). 2-1-2).
  • the assist depth map generation unit may set the first local region depth map by nonlinearly interpolating the vertex depth values of the first local region.
  • the assist depth map generation unit may set the second local area depth map by nonlinearly interpolating the vertex depth values of the second local area. That is, in this case, a local region depth map in which the depth value changes nonlinearly (curvilinearly) in the spatial direction is generated.
  • the manner in which the depth value changes in the spatial direction in this local region depth map may be in any manner as long as it is nonlinear.
  • the local region depth map (each depth value thereof) may be set so that the change in the depth value in the spatial direction is based on a nonlinear predetermined function. Further, for example, a user or the like may be allowed to set this function.
  • Method 1-2-1-3 when method 1-2-1 is applied, in the frame between key frames, as shown in the top row of the table in Figure 3, the vertex depth values of the local region depth map of the key frame are interpolated.
  • a vertex depth value of the local area depth map may be set, and other depth values of the interpolated local area depth map may be set based on the vertex depth value (method 1-2-1-3).
  • the assist depth map generation unit sets the vertex depth value of an interpolated local area that interpolates the vertex depth value of the first local area and the vertex depth value of the second local area, and uses the set vertex depth value.
  • An interpolated local area depth map may be set based on this.
  • the method for setting the interpolated local area depth map (other depth values) based on each vertex depth value in the interpolated local area is arbitrary.
  • the vertex depth values are linearly interpolated to obtain other depth values of the interpolated local area depth map.
  • other depth values of the interpolated local area depth map can be obtained by nonlinearly interpolating the vertex depth values. may be set.
  • the manner in which the depth value changes in the spatial direction in the interpolated local area depth map may be any type as long as it is nonlinear.
  • the interpolated local area depth map (each depth value thereof) may be set so that the change in the depth value in the spatial direction is based on a nonlinear predetermined function. Further, for example, a user or the like may be allowed to set this function.
  • the interpolation method of vertex depth values between frames is arbitrary.
  • the vertex depth values of the interpolated local region depth map may be set by linearly interpolating the vertex depth values of the local region depth map of the key frame ( Method 1-2-1-3-1).
  • the assist depth map generation unit may set the vertex depth value of the interpolated local region by linearly interpolating the vertex depth value of the first local region and the vertex depth value of the second local region.
  • the vertex depth values are interpolated using a method similar to method 1-1-1 (linear interpolation of depth values in local regions). Therefore, an interpolated local area depth map in which the vertex depth value changes linearly (linearly) in the time direction is generated.
  • an interpolated local region depth map (and other depth values) is set based on the vertex depth values linearly interpolated in this way (method 1-2-1-3).
  • the user can easily set a local region depth map in which the depth value (at least the vertex depth value) changes linearly in the time direction without having to set the local region depth map for every frame.
  • the depth value (at least the vertex depth value) can easily change linearly in the time direction for the basic depth map corresponding to the video image without the user having to manually generate an assist depth map for every frame.
  • An assisted depth map having a local region depth map can be generated. That is, it is possible to easily edit a local region in which the depth value (at least the vertex depth value) of the depth map changes linearly in the time direction.
  • the vertex depth values of the local region depth map of the key frame are interpolated by non-linear interpolation.
  • the vertex depth value of the local area depth map may be set (method 1-2-1-3-2).
  • the assist depth map generation unit may set the vertex depth value of the interpolated local area by nonlinearly interpolating the vertex depth value of the first local area and the vertex depth value of the second local area. That is, in this case, the vertex depth values are interpolated using a method similar to method 1-1-2 (nonlinear interpolation of depth values in local regions). Therefore, an interpolated local area depth map in which the vertex depth value changes nonlinearly (curvilinearly) in the time direction is generated.
  • an interpolated local region depth map (and other depth values) is set based on the vertex depth values linearly interpolated in this way (method 1-2-1-3).
  • the user can easily set a local area depth map in which the depth value (at least the vertex depth value) changes nonlinearly in the time direction without having to set the local area depth map for every frame. Can be done.
  • the depth value (at least the vertex depth value) can easily change non-linearly in the time direction without the user manually generating an assist depth map for every frame.
  • An assisted depth map having a local region depth map can be generated. That is, it is possible to easily edit a local region in which the depth value (at least the vertex depth value) of the depth map changes nonlinearly in the time direction.
  • the vertex depth value of the interpolated local area depth map may be set according to the value (method 1-2-1-3-3).
  • the reception unit accepts the designation of a reference object that refers to the depth value
  • the assist depth map generation unit generates the vertex depth value of the first local region, the vertex depth value of the second local region, and the interpolated local region.
  • the vertex depth value of may be set based on the depth value of the reference object of each frame.
  • the vertex depth values are interpolated using a method similar to method 1-1-3 (reference interpolation of depth values in local regions). Therefore, an interpolated local area depth map is generated in which the change in the vertex depth value in the time direction is similar to that of the reference object.
  • any object that appears in each frame of the RGB image can be set as this reference object.
  • an object that includes the local area may be used as the reference object, or an object that is located at a location different from the local area may be used as the reference object.
  • the vertex depth value of the local area depth map (including the interpolated local area depth map) of each frame may be set based on the depth value of the reference object, and any method may be used.
  • the vertex depth value of the local area depth map may be the same depth value as the reference object of the frame, or may be a value obtained by performing a predetermined calculation on the depth value of the reference object of the frame.
  • an interpolated local region depth map (and other depth values) is set based on the vertex depth values linearly interpolated in this way (method 1-2-1-3).
  • the user can easily set a local area depth map in which the change in the depth value (at least the vertex depth value) in the time direction is similar to that of the reference object, without the user having to set the local area depth map for every frame. can do.
  • the user can easily compare temporal changes in depth values (at least vertex depth values) with reference objects without having to manually generate assist depth maps for all frames.
  • An assisted depth map can be generated that has a local region depth map that is similar. That is, it is possible to easily edit a local region in which the depth value (at least the vertex depth value) of the depth map changes in the same way as the depth value of the reference object in the time direction.
  • the vertex depth value of the interpolated local region depth map may be set according to the setting (method 1-2-1-3-4).
  • the reception unit accepts the specification of the amount of change in the vertex depth value per frame
  • the assist depth map generation unit generates The vertex depth value of the interpolated local area may be set. That is, in this case, the vertex depth values are interpolated using a method similar to method 1-1-4 (manual interpolation of depth values in local regions). Therefore, an interpolated local area depth map in which the vertex depth value changes in the time direction according to the set amount of change is generated.
  • an interpolated local region depth map (and other depth values) is set based on the vertex depth values linearly interpolated in this way (method 1-2-1-3).
  • the user can easily set a local region depth map in which the depth value (at least the vertex depth value) changes by the desired amount of change in the time direction, without having to set the local region depth map for every frame. can do.
  • the user can easily change the depth value (at least the vertex depth value) to the desired amount of change in the time direction without having to manually generate an assist depth map for every frame with respect to the basic depth map corresponding to the video image.
  • An assisted depth map can be generated that has a varying local region depth map. That is, it is possible to easily edit a local region in which the depth value (at least the vertex depth value) of the depth map changes by a desired amount of change in the time direction.
  • an interpolated local region depth map may be generated using a neural network, as shown in the sixth row from the top of the table in FIG. (Method 1-2-1-3-5).
  • This neural network is a neural network generated by prior learning so as to output an interpolated local area depth map based on the information of the local area depth map of the key frame.
  • the assist depth map generation unit sets the interpolated local area depth map using a neural network that outputs the interpolated local area depth map based on the information of the first local area depth map and the second local area depth map. You may.
  • the vertex depth values are interpolated using a method similar to method 1-1-5 (AI interpolation of depth values in local regions). Therefore, an interpolated local region depth map is generated in which the vertex depth value changes in the temporal direction according to the specifications of the neural network.
  • the user can easily set a local region depth map in which the depth value (at least the vertex depth value) changes in the time direction without having to set the local region depth map for all frames.
  • the user can easily create a local region depth whose depth value (at least vertex depth value) changes in the time direction without having to manually generate an assist depth map for every frame.
  • An assisted depth map with a map can be generated. That is, it is possible to easily edit a local region where the depth value (at least the vertex depth value) of the depth map changes in the time direction.
  • a neural network generated by prior learning is used to output the vertex depth value of the interpolated local region depth map based on the vertex depth value of the local region depth map of the key frame. You may also generate vertex depth values for the map. In that case, in each frame between key frames, an interpolated local region depth map (other depth values) is set based on the vertex depth values linearly interpolated in this way (method 1-2-1-3). ).
  • a confirmation screen for confirming the local area depth map and the interpolated local area depth map may be displayed as shown in the seventh row from the top of the table in FIG. Method 1-3).
  • the information processing device may include a confirmation screen display control unit that generates and displays a confirmation screen for confirming the above-described first local region depth map, second local region depth map, and interpolated local region depth map. Further provision may be made.
  • the setting result of the dynamic local area depth map may be displayed on the display unit so that the user or the like can confirm it.
  • the confirmation screen 170 shown in FIG. 12 includes a display section 171 that displays the setting results of the local region depth map, and an operation section 172 that is a GUI (Graphical User Interface) for controlling the display of the display section 171.
  • GUI Graphic User Interface
  • the display unit 171 displays at least the assist depth map (the set local area depth map).
  • the assist depth map may be displayed superimposed on an RGB image (frame image of a moving image).
  • a composite image 181 in which a local region depth map 191 is superimposed on an RGB image as shown in the upper left of FIG. 13 may be displayed on the display unit 171.
  • This local area depth map 191 shows the set (interpolated) local area depth map in that frame, which is included in the assist depth map.
  • a basic depth map and an overwritten depth map may be further superimposed on the composite image 181.
  • the assist depth map may also be displayed superimposed on the base depth map.
  • a composite depth map 182 in which a local region depth map 191 is superimposed on the base depth map may be displayed on the display unit 171.
  • An overwrite depth map may also be superimposed on the composite depth map 182.
  • the assist depth map may also be superimposed on the overwrite depth map instead of the base depth map.
  • assist depth map 183 (local region depth map 191) may be displayed on the display unit 171, as shown in the lower center of FIG. 13.
  • one local region depth map (local region depth map 191) is set in the assist depth map 183, but a plurality of local region depth maps are set in the assist depth map 183.
  • a plurality of local area depth maps may be displayed on the display unit 171). The same applies to the composite image 181 and composite depth map 182.
  • the assist depth map corresponds to a moving image
  • it is displayed on the display unit 171 frame by frame.
  • the display unit 171 displays a frame of the assist depth map according to a user instruction or the like input using the GUI of the operation unit 172.
  • a user or the like may operate the operation unit 172 to specify a frame, and the specified frame of the assist depth map may be displayed on the display unit.
  • the user or the like operates the operation unit 172 to instruct playback of the entire sequence or a part of the sequence (normal speed playback, fast forward/rewind playback, frame-by-frame playback/frame-back playback, etc.), and assist depth map.
  • the specified section (the whole or part of the sequence) may be displayed in the specified playback method.
  • the confirmation screen display control unit may highlight the first local region depth map, the second local region depth map, and the interpolated local region depth map on the confirmation screen.
  • This highlighting method is arbitrary.
  • the (interpolated) local area depth map may be given a different color or pattern from other parts, or may be given a different density from other parts.
  • the outer frame of the (interpolated) local area depth map may be surrounded by a thick line or the like. Further, a pattern such as an arrow pointing to the (interpolated) local region depth map may be attached.
  • the user and the like can more easily identify the (interpolated) local area depth map on the display unit 171.
  • the user and the like can more easily confirm the setting results of the dynamic local area depth map.
  • Method 1-3 when applying Method 1-3, as shown in the ninth row from the top of the table in Figure 3, on this confirmation screen, the depth value (distribution) of the (interpolated) local area or (the distribution of the (interpolated) )
  • Each vertex depth value (or the distribution of depth values within the local area) of the local area depth map may be displayed (method 1-3-2).
  • method 1-1 described above that is, when the local region depth map is composed of a single depth value
  • the depth value is
  • the vertex depth value may be displayed on (the display unit 171 of) the confirmation screen 170 in FIG. 12 . Further, the distribution of depth values within the local area may be displayed.
  • the confirmation screen display control unit displays the depth values or vertex depth values (or distribution of depth values) of the first local region depth map, the second local region depth map, and the interpolated local region depth map on the confirmation screen.
  • the position of this display is arbitrary. For example, it may be displayed within the display section 171 or outside the display section 171. Further, this display mode may be of any type as long as mutually different depth values can be distinguished.
  • the depth value may be expressed numerically. Further, the depth value may be expressed using color, density, pattern, or the like.
  • the user or the like can more easily display the setting results ((distribution) of depth values, vertex depth values, etc.) of the (interpolated) local area depth map on the display unit 171.
  • the user and the like can more easily confirm the setting results of the dynamic local area depth map.
  • Method 1-3-3 when applying method 1-3, as shown at the bottom of the table in Figure 3, when displaying key frames on this confirmation screen, the local area and local area depth map may be editable. (Method 1-3-3). For example, when information about a first key frame or a second key frame is displayed on this confirmation screen, the reception unit may issue an editing instruction for the first local region and the first local region depth map, or a second key frame. An editing instruction for the local region and the second local region depth map may be received.
  • the user or the like can update (correct) the settings of the local area and the local area depth map on the confirmation screen based on the confirmed setting results.
  • the user or the like can more easily specify local areas, depth values, etc. so as to more easily obtain desired setting results.
  • a display for checking the local area depth map and the interpolated local area depth map may be displayed on the assist depth map editing screen (GUI for the user to specify the local area, depth value, etc.).
  • GUI assist depth map editing screen
  • the setting result of the local area may be displayed on the editing screen.
  • the editing screen may include the display section 171 and the operation section 172 shown in FIG. 12.
  • an assist depth map may be generated in which the local region changes in the time direction (method 1-4).
  • the shape of the first local region and the shape of the second local region described above may be different from each other. That is, when generating an assist depth map for a basic depth map corresponding to a moving image, the shape of the local region may be dynamic (transformed in the temporal direction).
  • the "shape" of the local region also includes the "size" of the local region. In other words, for example, if the local regions have different sizes, they can be said to have different shapes.
  • the user can easily set a local region depth map in which the shape of the local region is dynamic, without having to set the local region depth map for all frames.
  • the user can easily create an assisted depth map that has a local region depth map with a dynamic shape of the local region without having to manually generate an assisted depth map for every frame. can be generated. That is, it is possible to easily edit a local region of the depth map whose shape changes in the time direction.
  • each vertex of the local area is set in each key frame, and the local area is defined based on the A local area depth map may be set for the local area (method 1-4-1).
  • the reception unit accepts the designation of each vertex of the first local region and the designation of each vertex of the second local region, and the assist depth map generation unit generates a map based on the designated vertices of the first local region.
  • the first local area may be set based on the specified vertices of the second local area.
  • the user sets frame X of a moving image as a key frame, sets a local region in the assist depth map 121 of that frame X, and sets the local region depth map 131, as shown in FIG. .
  • the user specifies the vertices 131A, 131B, 131C, and 131D in the assist depth map 121, a local area having these as vertices is set, and the local area depth map 131 is set.
  • the user sets frame X+30 as a key frame, sets a local area corresponding to the local area of frame X in the assist depth map 123 of that frame Let's set it.
  • the user specifies the vertices 132A, 132B, 132C, and 132D in the assist depth map 123, a local area having these as vertices is set, and the local area depth map 132 is set.
  • local regions with mutually different shapes can be set for a plurality of key frames.
  • the user can easily set a local region depth map in which the shape of the local region is dynamic without setting the local region depth map for all frames.
  • the local area depth map for the set local area may be of any type.
  • this local region depth map may be composed of a single depth value. That is, one depth value may be set as a local area depth map for this set local area.
  • this local area depth map may be composed of a plurality of depth values. Then, as in the case of method 1-2-1 described above, a vertex depth value is set for this set local area, and other depth values of the local area depth map are set based on the vertex depth value. may be done.
  • vertices may be managed for each local area in the assist depth map.
  • specified vertices may be managed for each local area.
  • any method may be used to set the local area based on the specified vertex in the key frame.
  • the area between specified vertices may be the outer frame of the local area, and when the specified vertices are re-specified, the local area may be set ( Method 1-4-1-1).
  • the assist depth map generation unit may set the area between the specified vertices as the outer frame of the local area, and when the specified vertices are re-specified, the local area may be set.
  • FIG. 15 shows an example of how the local area is set.
  • the vertex 211 is set at the position indicated by the cursor 201.
  • the vertex 212 is set at that position.
  • a line segment 221 connecting the vertices 211 and 212 is set as the outer frame (edge) of the local area.
  • the apex 213 is set at that position. Furthermore, a line segment 222 connecting the vertices 212 and 213 is set as the outer frame (edge) of the local area. Thereafter, when the user moves the cursor 201 as indicated by the dotted arrow to specify a fourth vertex at a position different from the vertices 211 to 213, the vertex 214 is set at that position. Furthermore, a line segment 223 connecting the vertices 213 and 214 is set as the outer frame (edge) of the local area.
  • the line segment 224 connecting the vertex 214 and the vertex 211 is set as the outer frame (edge) of the local area. .
  • a closed area surrounded by edges is formed, and this area is set as the local area 220.
  • the user can easily set a local area with the desired number of vertices and shape simply by specifying the vertex positions.
  • a local area may be set (method 1-4-1-2).
  • the assist depth map generation unit may set the local area between specified vertices as the outer frame of the local area, and when a predetermined number of vertices are specified, the local area may be set.
  • FIG. 16 shows an example of how the local area is set.
  • the default number of vertices 231 is shown on the assist depth map editing screen.
  • this number of vertices is "4".
  • This value may be a fixed value or a value that can be set by the user or the like. Alternatively, it may be a value that can be set for each key frame. This value only needs to be set at least before specifying the vertex.
  • the vertex 211 is set at the position indicated by the cursor 201.
  • the vertex 212 is set at that position.
  • a line segment 221 connecting the vertices 211 and 212 is set as the outer frame (edge) of the local area.
  • the apex 213 is set at that position.
  • a line segment 222 connecting the vertices 212 and 213 is set as the outer frame (edge) of the local area. Thereafter, when the user moves the cursor 201 as indicated by the dotted arrow to specify a fourth vertex at a position different from the vertices 211 to 213, the vertex 214 is set at that position. Furthermore, a line segment 223 connecting the vertices 213 and 214 is set as the outer frame (edge) of the local area.
  • the number of vertices specified by the user has reached the predetermined number ("4"), so the line segment 224 connecting the vertex 214 and the first specified vertex 211 is set as the outer frame (edge) of the local area. Ru. As a result, a closed area surrounded by edges is formed, and this area is set as the local area 220.
  • the user can easily set a local area with a desired number of vertices and shape by simply specifying the number of vertices and the position of the vertices.
  • Method 1-4-1 when method 1-4-1 is applied, for example, as shown in the fifth row from the top of the table in Figure 4, the local area of the previous key frame is duplicated, and the local area is You may edit it (method 1-4-1-3).
  • the assist depth map generation unit may copy the local area of the previous key frame and update the position of the vertex of the local area based on the received instruction to change the position of the apex of the local area.
  • FIG. 17 shows an example of how the local area is set.
  • the local area local area depth map
  • the local area local area depth map
  • the position of each vertex (vertex 211 to vertex 214) of this local area can be changed.
  • the vertex 212 is set at a new position indicated by the cursor 201 (the position of the vertex 212 is updated).
  • the shape of the local region 220 changes (is updated). Note that in editing such a local area, it may be possible to add or delete vertices.
  • the user can more easily set a local region of a desired shape by using the local region set in the previous key frame.
  • Method 1-4-1 when method 1-4-1 is applied, that is, when local areas are set in key frames as described above, in the frames between the key frames, for example, 6 rows from the top of the table in FIG.
  • the position of the vertex of the local region of the key frame may be interpolated to set the position of the vertex of the interpolated local region (method 1-4-1-4). That is, in the interpolated local area, the shape of the local area (the position of the vertex of the local area) may be interpolated.
  • the assist depth map generation unit sets the vertices of an interpolated local area that interpolates the vertices of the first local area and the vertices of the second local area, and sets the interpolated local area based on the set vertices. You can.
  • the assist depth map generation unit sets the interpolation local region in the frames between the key frames (frame X and frame X+30).
  • the assist depth map 122 of frame The shape of the region depth map 131) and the shape of the local region (local region depth map 132) of the assist depth map 123 of frame X+30 are set to be interpolated. That is, the local area depth map 131 (the local area), the local area depth map 132 (the local area), and the interpolated local area depth map 133 (the interpolated local area) have different shapes.
  • any method may be used to interpolate the depth value (local region depth map).
  • one depth value is set as a local region depth map for a local region in a key frame (method 1-1)
  • one of methods 1-1-1 to 1-1-5 is used.
  • an interpolated local region depth map may be set. That is, for example, an interpolated local area depth map may be generated by linearly interpolating the depth value of the local area of the key frame (method 1-1-1). Alternatively, an interpolated local area depth map may be generated by nonlinearly interpolating the depth value of the local area of the key frame (method 1-1-2).
  • a reference object may be recognized in the RGB image, and an interpolated local region depth map may be generated according to the depth value of the reference object between key frames (method 1-1-3).
  • the amount of change in depth value per frame may be set in a key frame, and an interpolated local area depth map may be generated according to the setting (method 1-1-4).
  • an interpolated local region depth map may be generated using a neural network (method 1-1-5).
  • the method An interpolated local area depth map may be set according to 1-2-1-3. That is, the vertex depth values of the keyframe's local region depth map are interpolated to set the vertex depth values of the interpolated local region depth map, and other depth values of the interpolated local region depth map are set based on the vertex depth values. You can.
  • the vertex depth values of the interpolated local region depth map may be set by linearly interpolating the vertex depth values of the local region depth map of the key frame (method 1-2-1-3-1).
  • the vertex depth values of the interpolated local region depth map may be set by nonlinearly interpolating the vertex depth values of the local region depth map of the key frame (method 1-2-1-3-2).
  • the reference object may be recognized in the RGB image and the vertex depth value of the interpolated local area depth map may be set according to the depth value of the reference object between key frames (method 1-2-1-3-3).
  • the amount of change in vertex depth value per frame may be set in a key frame, and the vertex depth value of the interpolated local region depth map may be set in accordance with the setting (method 1-2-1-3-4).
  • the interpolated local area depth map may be generated using a neural network generated through prior learning so as to output the interpolated local area depth map based on the information of the local area depth map of the key frame ( Method 1-2-1-3-5).
  • other depth values may be set by linearly interpolating vertex depth values, as in method 1-2-1-1. Further, as in method 1-2-1-2, other depth values may be set by non-linear interpolation of the vertex depth values.
  • the interpolation method for (the position of) vertices between frames is arbitrary.
  • the vertices (positions) of the local areas of the key frames are linearly interpolated to may also be set (method 1-4-1-4-1).
  • the assist depth map generation unit linearly interpolates (the positions of) the vertices of the first local area and the vertices of the second local area, thereby determining (the positions of) the vertices of the interpolated local area. May be set.
  • the user can easily set a local region in which the position of the vertex changes linearly in the time direction, without having to set the local region for all frames.
  • the user can easily create an assist depth map that has a local area where the position of the vertex changes linearly in the time direction without having to manually generate an assist depth map for every frame.
  • a depth map can be generated. That is, it is possible to easily edit a local region of the depth map in which the position of the vertex changes linearly in the time direction.
  • the vertex (position) of the interpolated local area may be set by non-linear interpolation of the vertex (position) (method 1-4-1-4-2).
  • the assist depth map generation unit non-linearly interpolates the vertices (positions) of the first local area and the vertices (positions) of the second local area, thereby determining the vertices (positions) of the interpolated local area. May be set.
  • the vertices (positions of) of the local area are interpolated using a method similar to method 1-1-2 (nonlinear interpolation of depth values of local area). Therefore, an interpolated local region in which the position of the vertex changes nonlinearly (curvilinearly) in the time direction is generated.
  • the method for setting the position of the vertex of this interpolated local area is arbitrary.
  • the position of the vertex of the interpolated local area of each frame may be set based on a nonlinear predetermined function. Further, for example, a user or the like may be allowed to set this function. Further, the position of the vertex of the interpolated local area of each frame may be set to follow the movement of the entire screen (change in depth value).
  • the user can easily set a local region in which the position of the vertex changes non-linearly in the time direction, without having to set the local region for all frames.
  • the user can easily create an assist depth map that has a local area where the position of the vertex changes non-linearly in the time direction without having to manually generate an assist depth map for every frame.
  • a depth map can be generated. That is, it is possible to easily edit a local region of the depth map in which the position of the vertex changes nonlinearly in the time direction.
  • ⁇ Method 1-4-1-4-3> when method 1-4-1-4 is applied, in frames between key frames, for example, as shown in the ninth row from the top of the table in FIG. 4, a reference object is recognized in the RGB image,
  • the apex (position) of the interpolation local area may be set depending on the position of the reference object between key frames (method 1-4-1-4-3).
  • the reception unit accepts the designation of a reference object whose position is to be referred to, and the assist depth map generation unit generates (the position of) the vertex of the first local region, (the position of) the vertex of the second local region, and
  • the apex (position) of the interpolated local area may be set based on the position of the reference object in each frame.
  • the vertices (positions of) of the local area are interpolated using a method similar to method 1-1-3 (reference interpolation of depth value of local area). Therefore, an interpolated local region is generated in which the change in the position of the vertex in the time direction is similar to that of the reference object.
  • any object that appears in each frame of the RGB image can be set as this reference object.
  • an object that includes the local area may be used as the reference object, or an object that is located at a location different from the local area may be used as the reference object.
  • the user can easily set a local region where the change in the position of the vertex in the time direction is similar to that of the reference object, without the user having to set the local region for all frames.
  • the user can easily create local regions where the change in the position of the vertices over time is similar to that of the reference object, without having to manually generate an assisted depth map for every frame. It is possible to generate an assisted depth map having the following. That is, it is possible to easily edit a local region of the depth map in which the position of the vertex changes in the time direction in the same way as the reference object.
  • Method 1-4-1-4-4 when method 1-4-1-4 is applied, in the frames between key frames, for example, as shown in the 10th row from the top of the table in Figure 4, the number of vertices per frame in the key frame is The amount of change in position may be set, and (the position of) the vertex of the interpolated local area may be set in accordance with the setting (method 1-4-1-4-4).
  • the reception unit accepts the specification of the amount of change in the position of the vertex per frame, and the assist depth map generation unit generates The apex (position) of the interpolation local area may be set.
  • the user can easily set a local region in which the position of the vertex changes by a desired amount of change in the time direction, without having to set the local region for all frames.
  • the user can easily create a local region where the position of the vertex changes by a desired amount of change in the time direction, without having to manually generate an assist depth map for every frame. It is possible to generate an assisted depth map with That is, it is possible to easily edit a local region of the depth map in which the position of the vertex changes by a desired amount of change in the time direction.
  • This neural network is a neural network generated by prior learning so as to output an interpolated local area depth map based on the information of the local area of the key frame.
  • the assist depth map generation unit may set the interpolated local area depth map using a neural network that outputs the interpolated local area based on information on the first local area and the second local area. That is, in this case, (the position of) the vertex of the local area is interpolated using the same method as method 1-1-5 (AI interpolation of depth value of local area). Therefore, an interpolated local region depth map is generated in which the positions of vertices change in the time direction according to the specifications of the neural network.
  • the user can easily set a local region depth map in which the position of the vertex changes in the time direction, without having to set the local region depth map for all frames.
  • the user can easily create an assisted depth map that has a local area depth map in which the position of the vertex changes in the time direction, without having to manually generate an assisted depth map for every frame. Maps can be generated. That is, it is possible to easily edit a local region of the depth map in which the position of the vertex changes in the time direction.
  • a neural network generated through prior learning may be used to output the vertices (positions) of the interpolated local area based on the vertices (positions) of the local area of the key frame. You may also derive (the position of) the vertex. In that case, in each frame between key frames, an interpolated local region depth map of an interpolated local region having vertices interpolated in this way is set.
  • the number of (interpolated) local regions set in each frame of the assist depth map is arbitrary. In other words, one or more (interpolated) local regions can be set in each frame of the assist depth map.
  • the (interpolated) local regions may overlap each other. For example, part or all of the first local area may overlap with part or all of the second local area. Furthermore, three or more local regions may overlap each other.
  • the optimization processing device further includes an optimization processing unit that performs optimization processing using the basic depth map and the assist depth map, and the depth value of the assist depth map is adopted in the optimization processing, the optimization processing The section may select the depth value of one of the local regions in a region where a plurality of local regions overlap.
  • any method may be used to select this depth value. For example, a priority may be set for each local area, and the depth value of the local area with the highest priority may be selected. Alternatively, a depth value of a local area that is set later may be selected. Furthermore, the smallest depth value (that is, the closest one) among the depth values of the plurality of local regions may be selected. Alternatively, two or more of the depth values of the plurality of local regions may be combined using an arbitrary method, and the combined result may be selected.
  • FIG. 18 is a diagram illustrating a main configuration example of a computer that is an embodiment of an image processing device to which the present technology is applied.
  • a computer 300 shown in FIG. 18 is a device that installs and executes an application program (software) for editing depth data.
  • the computer includes a computer built into dedicated hardware and, for example, a general-purpose personal computer that can execute various functions by installing various programs.
  • the computer 300 executes, for example, ⁇ 3.
  • Assist Depth Map Having Dynamic Depth Value> the depth map (basic depth map) is edited using the various methods described above. By doing so, it is possible to generate a composite depth map (output depth map) obtained by editing the basic depth map.
  • FIG. 18 shows the main things such as the processing unit and the flow of data, and not all of the things shown in FIG. 18 are shown. That is, in the computer 300, there may be processing units that are not shown as blocks in FIG. 18, or processes or data flows that are not shown as arrows or the like in FIG. 18.
  • a CPU Central Processing Unit
  • ROM Read Only Memory
  • RAM Random Access Memory
  • An input/output interface 310 is also connected to the bus 304.
  • An input section 311 , an output section 312 , a storage section 313 , a communication section 314 , and a drive 315 are connected to the input/output interface 310 .
  • the input unit 311 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like.
  • the output unit 312 includes, for example, a monitor, a speaker, an output terminal, and the like.
  • the storage unit 313 includes, for example, a hard disk, a RAM disk, a nonvolatile memory, and the like.
  • the communication unit 314 includes, for example, a network interface.
  • the drive 315 drives a removable recording medium 321 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
  • the CPU 301 for example, loads an application program stored in the storage unit 313 into the RAM 303 via the input/output interface 310 and the bus 304, and executes the application program as described above. A series of processes are performed.
  • the RAM 303 also appropriately stores data necessary for the CPU 301 to execute various processes.
  • the application program executed by the computer 300 can be applied by being recorded on a removable recording medium 321 such as a package medium, for example.
  • the application program can be installed in the storage unit 313 via the input/output interface 310 by attaching the removable recording medium 321 to the drive 315.
  • the application program can also be provided via a wired or wireless transmission medium, such as a local area network, the Internet, or digital satellite broadcasting. In that case, this application program can be received by the communication unit 314 and installed in the storage unit 313.
  • this application program can also be installed in the ROM 302 or the storage unit 313 in advance.
  • FIG. 19 is a functional block diagram showing, as functional blocks, functions realized by executing this application program on this computer 300.
  • the computer 300 realizes the function of generating and editing a depth map from an image.
  • the computer 300 generates a depth map consisting of depth values of relatively high reliability parts from the image, as described in Patent Document 1, for example, and compares the reliability based on the depth map etc. By estimating the depth value of the low-target area, a depth map corresponding to the image is generated. By doing so, a more reliable depth map can be generated.
  • the computer 300 includes a GUI processing section 351, a file acquisition section 352, an edge depth generation section 353, an optimization processing section 354, a file output section 355, and an assist depth generation section. 356, an overwrite depth generation section 357, a display image generation section 358, and other functional blocks. Each functional block will be explained below.
  • the GUI processing unit 351 performs processing related to the GUI displayed on (the monitor of) the output unit 312. For example, when the input unit 311 is operated (for example, an instruction is input) based on the GUI for editing the depth map displayed on (the monitor of) the output unit 312, the GUI processing unit 351 Processing may be performed according to the accepted operation (instruction).
  • the configuration including the GUI processing section 351, the input section 311, and (the monitor of) the output section 312 is also referred to as a UI (User Interface).
  • the configuration including the GUI processing section 351 and the input section 311 is also referred to as a reception section.
  • the GUI processing unit 351 and the input unit 311 specify a first local region in a first key frame of a basic depth map that is acquired by a depth sensor and corresponds to a moving image, and specify a first local region corresponding to the first local region.
  • the file acquisition unit 352 performs processing related to file acquisition.
  • the file acquisition unit 352 may acquire a file of a moving image (for example, a moving image of a captured image) corresponding to the depth map to be edited.
  • the file acquisition unit 352 may acquire a moving image file input from outside the computer 300 via the input unit 311.
  • the file acquisition unit 352 may acquire a moving image file stored in the storage unit 313.
  • the file acquisition unit 352 may acquire a moving image file transmitted from another device via the communication unit 314.
  • the file acquisition unit 352 may read and acquire a moving image file recorded on the removable recording medium 321 via the drive 315. Further, the file acquisition unit 352 may supply the acquired moving image file to other functional blocks.
  • the edge depth generation unit 353 performs processing related to generation of an edge depth map.
  • the edge depth map is a depth map configured by depth values near the edges of the image.
  • the edge portion of an image is a portion where relatively reliable depth values can be obtained.
  • the edge depth generation unit 353 may derive a depth value near the edge from which a relatively reliable depth value can be obtained, and generate an edge depth map.
  • the edge depth generating section 353 uses the moving image file acquired by the file acquiring section 352 to generate an edge depth map corresponding to the moving image.
  • An edge depth map corresponding to a moving image is composed of edge depth maps corresponding to each frame of the moving image.
  • This edge depth map is the target of editing processing.
  • this edge depth map is the basic depth map.
  • an invalid value is set as a depth value in an area of the edge depth map that corresponds to a portion that is not an edge of the image (that is, the depth is unknown).
  • the edge depth generation section 353 may supply the generated edge depth map (basic depth map) to the optimization processing section 354 or the display image generation section 358.
  • the optimization processing unit 354 performs processing related to optimization processing. Optimization processing (also referred to as semi-global processing) estimates the depth value of the edge depth map (basic depth map) for areas where invalid values are set as depth values (that is, areas where the depth is unknown). It is processing.
  • This estimation method is arbitrary. For example, the method described in Patent Document 1 may be applied.
  • the optimization processing unit 354 may acquire the edge depth map generated by the edge depth generation unit 353. Further, the optimization processing unit 354 may acquire the assist depth map generated by the assist depth generation unit 356. Further, the optimization processing unit 354 may acquire the overwrite depth map generated by the overwrite depth generation unit 357.
  • the optimization processing unit 354 executes optimization processing, synthesizes (superimposes) a plurality of edge depth maps, assist depth maps, and overwriting depth maps, and creates a single-layer depth map (synthesized depth map). may be generated. Further, when adopting the depth value of the assist depth map in the optimization process, the optimization processing unit 354 may select the depth value of any one of the local regions in which a plurality of local regions overlap. Further, the optimization processing unit 354 may supply the single-layer depth map (synthesized depth map) to the file output unit 355, display image generation unit 358, etc. as a final output depth map.
  • the file output unit 355 performs processing related to file output.
  • the file output unit 355 may output the output depth map (synthetic depth map) generated by the optimization processing unit 354.
  • the file output unit 355 may output the output depth map to the outside of the computer 300 via the output unit 312.
  • the file output unit 355 may cause the storage unit 313 to store the output depth map.
  • the file output unit 355 may transmit the output depth map to another device via the communication unit 314. Further, the file output unit 355 may record the output depth map on the removable recording medium 321 via the drive 315.
  • the assist depth generation unit 356 performs processing related to generation of an assist depth map.
  • the assist depth map is a depth map composed of depth data to be added (synthesized) to the basic depth map, and is a depth map for editing the basic depth map.
  • the assist depth generation unit 356 may generate this assist depth map based on instructions from a user or the like received via the GUI processing unit 351 or the like.
  • the assist depth generation unit 356 generates ⁇ 3.
  • the assist depth map may be generated by applying the method described above in the section ⁇ Assist Depth Map Having Dynamic Depth Values''.
  • the assist depth generation unit 356 generates first specified depth data representing a first local area depth map, second specified depth data representing a second local area depth map, and a first key frame and a first specified depth data representing a second local area depth map.
  • An assisted depth map may be generated that includes interpolated depth data representing an interpolated local region depth map corresponding to an interpolated local region in at least one frame between two key frames.
  • the assist depth generation section 356 may supply the generated assist depth map to the optimization processing section 354 and the display image generation section 358.
  • the overwrite depth generation unit 357 performs processing related to generation of an overwrite depth map.
  • the overwriting depth map is a depth map configured with depth data that updates the depth value of the basic depth map, and is a depth map for editing the basic depth map.
  • the overwrite depth generation unit 357 may generate this overwrite depth map based on an instruction from a user or the like received via the GUI processing unit 351 or the like. Further, the overwrite depth generation section 357 may supply the generated overwrite depth map to the optimization processing section 354 and the display image generation section 358.
  • the display image generation unit 358 performs processing related to generation of a display image to be displayed on (the monitor of, etc.) the output unit 312. For example, the display image generation unit 358 may acquire the edge depth map (basic depth map) generated by the edge depth generation unit 353. Further, the display image generation unit 358 may obtain the output depth map (synthesized depth map) generated by the optimization processing unit 354. Further, the display image generation unit 358 may acquire the assist depth map generated by the assist depth generation unit 356. Further, the display image generation unit 358 may acquire the overwrite depth map generated by the overwrite depth generation unit 357.
  • the display image generation unit 358 may acquire the edge depth map (basic depth map) generated by the edge depth generation unit 353. Further, the display image generation unit 358 may obtain the output depth map (synthesized depth map) generated by the optimization processing unit 354. Further, the display image generation unit 358 may acquire the assist depth map generated by the assist depth generation unit 356. Further, the display image generation unit 358 may acquire the overwrite depth map generated
  • the display image generation unit 358 may use the acquired depth map to generate a GUI (depth editing screen to be described later) for editing the depth map. Then, the display image generation unit 358 may supply the generated GUI (depth editing screen) as a display image to the output unit 312 and display it on the monitor.
  • GUI depth editing screen
  • the display image generation unit 358 generates a display image (depth editing screen including the superimposed image) using a superimposed image in which an edge depth map (basic depth map) and an RGB image corresponding to the edge depth map are superimposed. You can. Note that the display image generation unit 358 can also further superimpose an assist depth map or an overwrite depth map on the superimposed image. In other words, the display image generation section 358 can function as a superimposition processing section.
  • the display image generation unit 358 may generate a confirmation screen for confirming the setting results of the (interpolated) local region of the assist depth map, its (interpolated) local region depth map, etc., for example.
  • the display image generation section 358 can function as a confirmation screen display control section that generates and displays the confirmation screen.
  • the assist depth generation section 356 includes a key frame setting section 361, a local region depth map generation section 362, and an interpolated local region depth map generation section 363.
  • the key frame setting unit 361 performs processing related to setting key frames.
  • a key frame is a frame in which a user specifies a local region, local region depth map, etc. of an assist depth map corresponding to a moving image.
  • the key frame setting unit 361 may set this key frame for the assist depth map corresponding to the moving image based on a user instruction received via the input unit 311 or the like.
  • the local region depth map generation unit 362 performs processing related to the generation of a local region depth map. For example, the local area depth map generation unit 362 generates local area and A local region depth map may also be set. At this time, the local area depth map generation unit 362 performs ⁇ 3. Assist depth map having dynamic depth value> The method described above may be applied to set a local region and its local region depth map.
  • the interpolated local area depth map generation unit 363 performs processing related to generation of an interpolated local area depth map. For example, the interpolated local region depth map generation unit 363 interpolates the local regions of (previous and subsequent) key frames and their local region depth maps in frames between key frames of the assist depth map, and interpolates the interpolated local regions and their interpolated local regions. A depth map may also be generated. At this time, the interpolated local region depth map generation unit 363 performs the following steps: ⁇ 3. An interpolated local region and its interpolated local region depth map may be generated by applying the method described above in "Assist Depth Map Having Dynamic Depth Value.”
  • the local region depth map generation section 362 includes a local region setting section 371 and a depth value setting section 372.
  • the local area setting unit 371 performs processing related to local area setting. For example, the local area setting unit 371 sets a local area in the key frame set by the key frame setting unit 361 of the assist depth map based on an instruction from a user or the like received via the input unit 311 or the like. Good too. At this time, the local area setting unit 371 performs ⁇ 3.
  • the local area may be set by applying the method described above in ⁇ Assist Depth Map with Dynamic Depth Value''.
  • the depth value setting unit 372 performs processing related to setting a depth value (that is, a local region depth map) for the local region. For example, the depth value setting unit 372 sets the local area setting unit 371 based on an instruction from a user or the like received via the input unit 311 or the like in the key frame set by the key frame setting unit 361 of the assist depth map. A depth value (local region depth map) may be set for the local region set by . At this time, the depth value setting unit 372 selects ⁇ 3. The depth value (local area depth map) may be set by applying the method described above in ⁇ Assist Depth Map Having Dynamic Depth Value''.
  • FIG. 19 shows an example of a GUI for editing the depth map generated by the display image generation unit 358 in FIG. 18 and displayed on the output unit 312.
  • a depth editing screen 400 shown in FIG. 19 is an example of a GUI for editing this depth map.
  • the depth editing screen 400 includes a depth map display section 401, a layer editing section 402, and an optimization control section 403.
  • the depth map display section 401 displays a depth map to be edited (a preview of the edited depth map).
  • the depth map display unit 401 can also display (transparently display) the depth maps of each layer (or the depth maps of some layers) in a superimposed manner.
  • the depth map display unit 401 can also display the RGB image and the depth map in a superimposed manner (transparently display).
  • the RGB image is an image corresponding to a depth map, which is not a depth map, and is made up of luminance components, color components, etc., for example. For example, it is a captured image.
  • the RGB image may be a RAW image or a YUV image.
  • the image may be a color image or a monochrome image. Note that if the image is a RAW image or a YUV image, it is converted to an RGB image before being displayed on the depth map display section 401.
  • the layer editing section 402 is an area where operations are performed for each layer on which editing work is performed.
  • the layer editing section 402 has a layer selection field 411 and a timeline 412.
  • the layer selection field 411 is a GUI for selecting (designating) a layer for editing.
  • the timeline 412 is a GUI in which a sequence of moving images is shown horizontally. On this timeline 412, a frame to be processed is selected, for example, by specifying the position (horizontal position) of the pointer 413. In this way, frames selected using the timeline 412 of the depth map of the layer selected using the layer selection field 411 are displayed on the depth map display section 401.
  • the optimization control unit 403 is a GUI for optimization processing.
  • the optimization control unit 403 is provided with an execution button 431 and file name input fields 432 to 434.
  • the file name input field 432 the file name of the overwrite depth map used for the optimization process is input.
  • the file name input field 433 the file name of the assist depth map used for the optimization process is input.
  • the file name input field 434 the file name of the edge depth map used for the optimization process is input.
  • the configuration shown in the display image generation unit 358 in FIG. 18 is an example, and the configuration of the GUI for editing the depth map is not limited to this example, and may be of any type.
  • a key frame designation button or the like may be provided as a GUI for designating a key frame.
  • the user can easily set a dynamic local area depth map without setting local area depth maps for all frames. That is, the computer 300 can facilitate generation of an assist depth map having a dynamic local region depth map. Therefore, the user can easily edit a local region where the depth value of the depth map changes in the time direction.
  • the computer 300 executes depth map generation processing when editing the depth map. An example of the flow of this depth map generation process will be described with reference to the flowchart of FIG. 20.
  • the file acquisition unit 352 reads a file and acquires an RGB image (moving image) in step S301.
  • the edge depth generation unit 353 generates an edge depth map corresponding to each frame of the moving image.
  • step S303 the assist depth generation unit 356 determines whether a frame has been selected. If it is determined that the frame has been selected, the process advances to step S304. Further, in step S304, the key frame setting unit 361 determines whether or not to set the frame as a key frame. If it is determined to be a key frame, the process advances to step S305.
  • step S305 the key frame setting unit 361 sets the frame as a key frame.
  • step S306 the local region depth map generation unit 362 executes local region depth map generation processing, and generates a local region and its local region depth map in the key frame. At this time, the local area depth map generation unit 362 performs ⁇ 3.
  • Assisted Depth Map with Dynamic Depth Value> The method described above may be applied to generate a local region and its local region depth map.
  • step S306 Upon completion of the process in step S306, the process proceeds to step S307. Furthermore, if it is determined in step S304 that the frame is not to be used as a key frame, the process advances to step S307.
  • step S307 the overwrite depth generation unit 357 determines whether to edit the overwrite depth map. If it is determined that the overwrite depth map is to be edited, the process advances to step S308. In step S308, the overwrite depth generation unit 357 edits the overwrite depth map based on user instructions and the like.
  • step S308 Upon completion of the process in step S308, the process proceeds to step S309. Furthermore, if it is determined in step S307 that the overwrite depth map is not to be edited, the process advances to step S309. Furthermore, if it is determined in step S303 that no frame is selected, the process advances to step S309.
  • step S309 the assist depth generation unit 356 determines whether to finish editing the depth map. If it is determined that editing of the depth map is to be continued, the process returns to step S303, and the subsequent processes are repeated. Furthermore, if it is determined in step S309 that editing of the depth map is to be completed, the process proceeds to step S310.
  • step S310 the interpolated local area depth map generation unit 363 determines whether there are multiple key frames. If it is determined that it exists, the process advances to step S311.
  • step S311 the interpolated local area depth map generation unit 363 generates an interpolated local area and its interpolated local area depth map in each frame between key frames. At this time, the interpolated local region depth map generation unit 363 performs the following steps: ⁇ 3.
  • An interpolated local area and its interpolated local area depth map may be generated by applying the method described above in ⁇ Assist Depth Map Having Dynamic Depth Values''.
  • step S311 Upon completion of the process in step S311, the process proceeds to step S312. If it is determined in step S310 that there are no multiple key frames, the process proceeds to step S312.
  • step S312 the optimization processing unit 354 performs optimization processing and generates an output depth map with a single layer structure.
  • step S313 the file output unit 355 outputs the generated output depth map.
  • step S331 the GUI processing unit 351 and the input unit 311 accept the designation of vertices by the user or the like. Further, the local area setting unit 371 sets a local area having the specified vertex. At this time, the local area setting unit 371 performs ⁇ 3. Assist Depth Map with Dynamic Depth Value> A local area is set by applying the method described above.
  • step S332 the GUI processing unit 351 and the input unit 311 accept designation of a depth value for the local area or vertex by the user or the like. Further, the depth value setting unit 372 generates a local region depth map based on the specified depth value. At this time, the depth value setting unit 372 selects ⁇ 3. Assist Depth Map Having Dynamic Depth Value> A local region depth map is set by applying the method described above.
  • step S333 the local area depth map generation unit 362 determines whether to end the local area depth map generation process. If it is determined that the local area depth map generation process is not to end, that is, to set another local area, the process returns to step S331 and the subsequent processes are repeated. Further, if it is determined in step S333 that the local area depth map generation process is to be ended, the process returns to FIG. 20.
  • the user can easily set a dynamic local area depth map without setting local area depth maps for all frames. That is, the computer 300 can facilitate generation of an assist depth map having a dynamic local region depth map. Therefore, the user can easily edit a local region where the depth value of the depth map changes in the time direction.
  • the present technology can be applied to any configuration.
  • the present technology can be applied to various electronic devices.
  • the present technology can be applied to a processor (e.g., video processor) as a system LSI (Large Scale Integration), a module (e.g., video module) that uses multiple processors, etc., a unit (e.g., video unit) that uses multiple modules, etc.
  • a processor e.g., video processor
  • the present invention can be implemented as a part of a device, such as a set (for example, a video set), which is a unit with additional functions.
  • the present technology can also be applied to a network system configured by a plurality of devices.
  • the present technology may be implemented as cloud computing in which multiple devices share and jointly perform processing via a network.
  • this technology will be implemented in a cloud service that provides services related to images (moving images) to any terminal such as a computer, AV (Audio Visual) equipment, mobile information processing terminal, IoT (Internet of Things) device, etc. You may also do so.
  • a system refers to a collection of multiple components (devices, modules (components), etc.), and it does not matter whether all the components are in the same housing or not. Therefore, multiple devices housed in separate casings and connected via a network, and one device with multiple modules housed in one casing are both systems. .
  • Systems, devices, processing units, etc. to which this technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. . Moreover, its use is also arbitrary.
  • the configuration described as one device (or processing section) may be divided and configured as a plurality of devices (or processing sections).
  • the configurations described above as a plurality of devices (or processing units) may be configured as one device (or processing unit).
  • part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit) as long as the configuration and operation of the entire system are substantially the same. .
  • the above-mentioned program may be executed on any device.
  • the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.
  • each step of one flowchart may be executed by one device, or may be executed by multiple devices.
  • the multiple processes may be executed by one device, or may be shared and executed by multiple devices.
  • multiple processes included in one step can be executed as multiple steps.
  • processes described as multiple steps can also be executed together as one step.
  • the processing of the steps described in the program may be executed chronologically in the order described in this specification, or may be executed in parallel, or may be executed in parallel. It may also be configured to be executed individually at necessary timings, such as when a request is made. In other words, the processing of each step may be executed in a different order from the order described above, unless a contradiction occurs. Furthermore, the processing of the step of writing this program may be executed in parallel with the processing of other programs, or may be executed in combination with the processing of other programs.
  • the present technology can also have the following configuration. (1) designation of a first local region in a first key frame of a basic depth map acquired by a depth sensor and corresponding to a moving image; designation of a first local region depth map corresponding to the first local region; a reception unit that receives designation of a second local region in a second key frame of a basic depth map and designation of a second local region depth map corresponding to the second local region; first designated depth data representing the first local region depth map, second designated depth data representing the second local region depth map, and the first key frame and the second key frame.
  • an assist depth map generation unit that generates an assist depth map including interpolated depth data representing an interpolated local region depth map corresponding to an interpolated local region in at least one frame between;
  • the first local region depth map and the second local region depth map have different depth values,
  • the interpolation local area is a local area that interpolates the first local area and the second local area,
  • the interpolated local area depth map is a depth map that interpolates the first local area depth map and the second local area depth map.
  • Information processing apparatus (2) The information processing device according to (1), wherein the first local area depth map, the second local area depth map, and the interpolated local area depth map are each configured with a single depth value. .
  • the assist depth map generation unit generates the interpolated local area depth map by linearly interpolating the depth value of the first local area depth map and the depth value of the second local area depth map.
  • the information processing device according to (2) or (3) The assist depth map generation unit generates the interpolated local area depth map by nonlinearly interpolating the depth value of the first local area depth map and the depth value of the second local area depth map.
  • the information processing device according to (2) or (3) wherein the depth value is set.
  • the reception unit accepts a designation of a reference object that refers to a depth value
  • the assist depth map generation unit converts the depth value of the first local region depth map, the depth value of the second local region depth map, and the depth value of the interpolated local region depth map into respective The information processing device according to (2) or (3), wherein the information processing device is set based on a depth value of the reference object of a frame.
  • the reception unit accepts a designation of an amount of change in depth value per frame, In (2) or (3), the assist depth map generation unit sets the depth value of the interpolated local area depth map based on the depth value and the amount of change of the first local area depth map.
  • the assist depth map generation unit sets a depth value of the interpolated local area depth map using a neural network that outputs the interpolated local area depth map based on information on the local area of the key frame. ) or the information processing device according to (3).
  • the first local region depth map, the second local region depth map, and the interpolated local region depth map are each composed of a plurality of depth values (1) to (8).
  • the reception unit receives designation of each vertex depth value of the first local region and designation of each vertex depth value of the second local region,
  • the assist depth map generation unit sets the first local region depth map based on the vertex depth value of the specified first local region, and sets the first local region depth map based on the vertex depth value of the designated second local region.
  • the information processing device according to (9), wherein the second local region depth map is set based on the value.
  • (11) The information processing device according to (10), wherein in the first specified depth data, the second specified depth data, and the interpolated depth data, the vertex depth value is managed for each local area.
  • the assist depth map generation unit sets the first local region depth map by linearly interpolating the vertex depth values of the first local region, and sets the first local region depth map by linearly interpolating the vertex depth values of the second local region.
  • the information processing device according to (10) or (11), wherein the second local area depth map is set by linearly interpolating values.
  • the assist depth map generation unit sets the first local region depth map by non-linearly interpolating the vertex depth values of the first local region, and sets the vertex depth map of the second local region.
  • the information processing device according to (10) or (11), wherein the second local area depth map is set by nonlinearly interpolating values.
  • the assist depth map generation unit sets a vertex depth value of the interpolated local area that interpolates the vertex depth value of the first local area and the vertex depth value of the second local area, The information processing device according to (10) or (11), wherein the interpolated local area depth map is set based on the set vertex depth value.
  • the assist depth map generation unit linearly interpolates the vertex depth value of the first local region and the vertex depth value of the second local region, thereby generating the vertex depth of the interpolated local region.
  • the information processing device according to (14), in which a value is set.
  • the assist depth map generation unit calculates the vertex depth of the interpolated local region by non-linearly interpolating the vertex depth value of the first local region and the vertex depth value of the second local region.
  • the information processing device in which a value is set.
  • the reception unit receives a designation of a reference object that refers to a depth value
  • the assist depth map generation unit converts the vertex depth value of the first local region, the vertex depth value of the second local region, and the vertex depth value of the interpolated local region into the vertex depth values of the respective frames.
  • the information processing device according to (14) wherein the information processing device is set based on the depth value of the reference object.
  • the accepting unit accepts a designation of an amount of change in the vertex depth value per frame, The information processing device according to (14), wherein the assist depth map generation unit sets the vertex depth value of the interpolated local area based on the vertex depth value of the first local area and the amount of change.
  • the assist depth map generation unit generates the interpolated local area depth map using a neural network that outputs the interpolated local area depth map based on information of the first local area depth map and the second local area depth map.
  • the information processing device according to (14), which sets a local region depth map.
  • the information processing device causes the first local region depth map, the second local region depth map, and the interpolated local region depth map to be highlighted on the confirmation screen (20). information processing equipment.
  • the confirmation screen display control unit displays depth values or vertex depth values of the first local region depth map, the second local region depth map, and the interpolated local region depth map on the confirmation screen.
  • the information processing device according to (20) or (21).
  • the reception unit edits the first local area and the first local area depth map.
  • the information processing device according to any one of (20) to (22), wherein the information processing device receives an instruction or an editing instruction for the second local region and the second local region depth map.
  • the information processing device according to any one of (1) to (23), wherein the first local area and the second local area have different shapes.
  • the receiving unit receives a designation of each vertex of the first local region and a designation of each vertex of the second local region,
  • the assist depth map generation unit sets the first local area based on the apex of the specified first local area, and sets the first local area based on the apex of the specified second local area.
  • the information processing device according to (24), wherein the local area of 2 is set.
  • the information processing device according to (25), wherein in the first specified depth data, the second specified depth data, and the interpolated depth data, the vertices are managed for each local area.
  • the assist depth map generation unit sets the area between the specified vertices as an outer frame of the local area, and sets the local area when the specified vertices are re-designated. (25) or (26) ).
  • the assist depth map generation unit sets the area between the specified vertices as an outer frame of the local area, and sets the local area when a predetermined number of the vertices are specified.
  • the assist depth map generation unit copies the local area of the previous key frame, and updates the position of the apex of the local area based on the received instruction to change the position of the apex of the local area.
  • the information processing device according to (25) or (26).
  • the assist depth map generation unit sets vertices of the interpolation local area for interpolating the vertices of the first local area and the vertices of the second local area, and based on the set vertices,
  • the information processing device according to any one of (25) to (29), wherein the interpolation local area is set by (31)
  • the assist depth map generation unit linearly interpolates the position of the vertex of the first local region and the position of the vertex of the second local region, thereby determining the position of the vertex of the interpolated local region.
  • the information processing device according to (30) which sets a position.
  • the assist depth map generation unit is configured to non-linearly interpolate the position of the apex of the first local area and the position of the apex of the second local area, thereby determining the position of the apex of the interpolated local area.
  • the information processing device according to (30), which sets a position.
  • the reception unit receives a designation of a reference object whose position is to be referred to;
  • the assist depth map generation unit converts the positions of the vertices of the first local area, the positions of the vertices of the second local area, and the positions of the vertices of the interpolated local area into the positions of the vertices of each frame.
  • the information processing device wherein the information processing device is set based on the position of the reference object.
  • the receiving unit receives a designation of the amount of change in the position of the vertex per frame.
  • the assist depth map generation unit sets the position of the vertex of the interpolation local region based on the position of the vertex of the first local region and the amount of change.
  • the assist depth map generation unit sets the interpolated local area using a neural network that outputs the interpolated local area based on information on the first local area and the second local area. 30).
  • the information processing device according to item 30).
  • the optimization processing unit selects a depth value of one of the local regions in an area where the plurality of local regions overlap (1)
  • (37) specifying a first local area in a first key frame of a basic depth map acquired by a depth sensor and corresponding to a moving image; specifying a first local area depth map corresponding to the first local area; Accepting the designation of a second local region in a second key frame of the basic depth map and the designation of a second local region depth map corresponding to the second local region, first designated depth data representing the first local region depth map, second designated depth data representing the second local region depth map, and the first key frame and the second key frame.
  • an assisted depth map including interpolated depth data representing an interpolated local region depth map corresponding to the interpolated local region in at least one frame between;
  • the first local region depth map and the second local region depth map have different depth values,
  • the interpolation local area is a local area that interpolates the first local area and the second local area,
  • the interpolated local area depth map is a depth map that interpolates the first local area depth map and the second local area depth map.
  • a computer specifying a first local area in a first key frame of a basic depth map acquired by a depth sensor and corresponding to a moving image; specifying a first local area depth map corresponding to the first local area; and specifying the basic depth map a reception unit that accepts specification of a second local area in a second key frame of and specification of a second local area depth map corresponding to the second local area; first designated depth data representing the first local region depth map, second designated depth data representing the second local region depth map, and the first key frame and the second key frame.
  • the function as an assist depth map generation unit that generates an assist depth map including interpolated depth data representing an interpolated local area depth map corresponding to an interpolated local area depth map in at least one frame between;
  • the first local region depth map and the second local region depth map have different depth values,
  • the interpolation local area is a local area that interpolates the first local area and the second local area,
  • the interpolated local area depth map is a depth map that interpolates the first local area depth map and the second local area depth map.
  • 300 Computer 351 GUI processing unit, 352 File acquisition unit, 353 Edge depth generation unit, 354 Optimization processing unit, 355 File output unit, 356 Assist depth generation unit, 357 Overwrite depth generation unit, 358 Display image generation unit, 361 key frame Setting unit, 362 Local area depth map generation unit, 363 Interpolated local area depth map generation unit, 371 Local area setting unit, 372 Depth value setting unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Processing Or Creating Images (AREA)

Abstract

La présente divulgation concerne un dispositif de traitement d'informations, un procédé et un programme qui permettent d'éditer une région locale dans laquelle la valeur de profondeur d'une carte de profondeur change dans une direction temporelle. Dans la présente divulgation, la désignation d'une première carte de profondeur de région locale dans une première trame clé et la désignation d'une seconde carte de profondeur de région locale dans une seconde trame clé sont acceptées, et une carte de profondeur d'assistance qui comprend de premières données de profondeur désignées, de secondes données de profondeur désignées et des données de profondeur d'interpolation est générée. Les valeurs de profondeur des première et seconde cartes de profondeur de région locale sont différentes l'une de l'autre. La présente divulgation peut être appliquée, par exemple, à un dispositif de traitement d'informations, à un ordinateur, à un procédé de traitement d'informations ou à un programme.
PCT/JP2023/031088 2022-09-12 2023-08-29 Dispositif, procédé et programme de traitement d'informations WO2024057902A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022-144744 2022-09-12
JP2022144744 2022-09-12

Publications (1)

Publication Number Publication Date
WO2024057902A1 true WO2024057902A1 (fr) 2024-03-21

Family

ID=90275083

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/031088 WO2024057902A1 (fr) 2022-09-12 2023-08-29 Dispositif, procédé et programme de traitement d'informations

Country Status (1)

Country Link
WO (1) WO2024057902A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004229093A (ja) * 2003-01-24 2004-08-12 Nippon Telegr & Teleph Corp <Ntt> 立体画像生成方法及び立体画像生成装置、ならびに立体画像生成プログラム及び記録媒体
WO2012056686A1 (fr) * 2010-10-27 2012-05-03 パナソニック株式会社 Dispositif d'interpolation d'image 3d, dispositif d'imagerie 3d, et procédé d'interpolation d'image 3d
JP2014192794A (ja) * 2013-03-28 2014-10-06 Dainippon Printing Co Ltd 奥行き制作支援装置、奥行き制作方法、及びプログラム
JP2019114842A (ja) * 2017-12-21 2019-07-11 株式会社ソニー・インタラクティブエンタテインメント 画像処理装置、コンテンツ処理装置、コンテンツ処理システム、および画像処理方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004229093A (ja) * 2003-01-24 2004-08-12 Nippon Telegr & Teleph Corp <Ntt> 立体画像生成方法及び立体画像生成装置、ならびに立体画像生成プログラム及び記録媒体
WO2012056686A1 (fr) * 2010-10-27 2012-05-03 パナソニック株式会社 Dispositif d'interpolation d'image 3d, dispositif d'imagerie 3d, et procédé d'interpolation d'image 3d
JP2014192794A (ja) * 2013-03-28 2014-10-06 Dainippon Printing Co Ltd 奥行き制作支援装置、奥行き制作方法、及びプログラム
JP2019114842A (ja) * 2017-12-21 2019-07-11 株式会社ソニー・インタラクティブエンタテインメント 画像処理装置、コンテンツ処理装置、コンテンツ処理システム、および画像処理方法

Similar Documents

Publication Publication Date Title
KR101969864B1 (ko) 객체영상과 배경영상의 상호 연동에 기반한 영상 합성 방법
JP4591576B2 (ja) 画像処理装置、画像処理方法、プログラム
US9749601B2 (en) Imaging device, image display method, and storage medium for displaying reconstruction image
US5727093A (en) Image processing method and apparatus therefor
JP4624248B2 (ja) 画像処理装置、肌色調整方法及びプログラム
US8373802B1 (en) Art-directable retargeting for streaming video
JP2014165764A (ja) 編集装置、編集方法、及び編集プログラム
JP2017076409A (ja) シーン基準のメタデータ捕捉のための基準カード
JP4767258B2 (ja) 合成画像を生成する方法
KR101875047B1 (ko) 사진측량을 이용한 3d 모델링 시스템 및 방법
US20240046430A1 (en) Corrective Lighting for Video Inpainting
KR20210111733A (ko) 영상을 처리하는 방법, 장치 및 기록매체
US20130278602A1 (en) Animatable graphics lighting analysis reporting
CN114640885A (zh) 视频插帧方法、训练方法、装置和电子设备
WO2024057902A1 (fr) Dispositif, procédé et programme de traitement d&#39;informations
US20130136342A1 (en) Image processing device and image processing method
KR101819984B1 (ko) 실시간 영상 합성 방법
KR101632514B1 (ko) 깊이 영상 업샘플링 방법 및 장치
US9460544B2 (en) Device, method and computer program for generating a synthesized image from input images representing differing views
JP6930091B2 (ja) 画像処理装置、画像処理方法、画像処理システムおよびプログラム
WO2015029613A1 (fr) Dispositif de génération d&#39;images et dispositif d&#39;affichage
JP5093317B2 (ja) 画像処理装置、画像処理方法、プログラム
WO2021200190A1 (fr) Dispositif et procédé de traitement d&#39;informations, et programme
JP4826857B2 (ja) 画像処理方法、画像処理装置及び画像処理プログラム
JP5857606B2 (ja) 奥行き製作支援装置、奥行き製作支援方法、およびプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23865259

Country of ref document: EP

Kind code of ref document: A1