JP2019008582A - Video processing device, video processing method, and video processing program - Google Patents

Abstract

To realize speed processing of connecting a plurality of video data in a vertical direction and a horizontal direction.SOLUTION: A video processing device 1 includes frame acquiring means 22 for acquiring a plurality of frame data shot simultaneously from a plurality of video data; a region setting unit 23 for setting a longitudinally overlapping region where frame data adjacent only in a longitudinal direction overlaps from overlapping frame data where a plurality of frame data are overlapped with each other by matching feature points with respect to adjacent frame data, a transversely overlapping region where frame data adjacent only in a transverse direction overlaps, and a cross overlapping region where frame data adjacent in the longitudinal direction and the transverse direction overlap; seam calculating means 24 for calculating joints between frame data adjacent to each other in respective overlapping regions of the longitudinally overlapping region, the transversely overlapping region, and the cross overlapping region; and connected frame outputting means 26 for connecting each piece of frame data according to the joint calculated by the seam calculating means 24 and outputting the connected frame data without overlap.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a video processing apparatus, a video processing method, and a video processing program that connect a plurality of video data that are captured by overlapping a part of a shooting area adjacent to each other in the vertical and horizontal directions.

  In recent years, image processing in which video data captured by a plurality of cameras is connected and reconstructed as one panoramic video or omnidirectional video has become widespread. In such a technique, it is possible to output a powerful video on a large screen, and thus improvement in video quality is required (see, for example, Non-Patent Document 1). Non-Patent Document 1 discloses that the beauty of a video is improved by applying a reliability mapping method for representing a subject in a multilayer plane and two-dimensional image processing, regarding a technique for panoramic synthesis of images.

  However, with the improvement of video quality, calculation cost in image processing becomes a burden, and reduction of calculation cost is expected. In the technique described in Non-Patent Document 1, the computer load may become very high and the processing speed may decrease.

  Specifically, when processing a plurality of 4K videos (3840 × 2160p, 60 fps), the amount of data that must be processed per second per video reaches 6 Gbits. If it is desired to connect video shot by a plurality of cameras and perform live distribution, the image processing system may be heavily loaded, and processing time may be required so that live distribution cannot be realized.

  Therefore, in order to ensure the practicality of live distribution, a technique for realizing high-speed processing (real-time processing for live distribution) in addition to the naturalness of joints is disclosed (see Non-Patent Document 2). .

Ai Isogai, Yutaka Kunida, Hideaki Kizen, “Proposal of panoramic video generation technology adapted to image areas”, IPSJ SIG Technical report Vol. 2009-AVM-66 No.9 2009/9/24 Yumi Kikuchi, Akira Ono, “Real-time generation and evaluation of 4K × N panorama content”, IEICE Technical Report, vol. 116, no. 73, MVE2016-2, pp. 23-28, June 2016

  However, although the above-mentioned Non-Patent Documents 1 and 2 refer to panoramic video in which video data is connected only in the horizontal direction, the video processing when the video data is connected not only in the horizontal direction but also in the vertical direction is sufficient. Not considered.

  Therefore, the present invention is to provide a video processing apparatus, a video processing method, and a video processing program capable of realizing high-speed processing for connecting a plurality of video data in the vertical direction and the horizontal direction.

  In order to solve the above-described problem, a first feature of the present invention relates to a video processing apparatus that connects a plurality of video data that are adjacent to each other in a vertical direction and a horizontal direction and that are captured by overlapping a part of a shooting area. A video processing apparatus according to a first feature of the present invention includes a frame acquisition unit that acquires a plurality of frame data shot simultaneously from a plurality of video data, and a plurality of feature points that coincide with each other in adjacent frame data. From the overlapped frame data that overlaps the frame data, the vertical overlap area where the frame data adjacent in the vertical direction only overlaps, the horizontal overlap area where the frame data adjacent only in the horizontal direction overlaps, and the vertical and horizontal adjacent Area setting means for setting a cross overlap area where frame data to be overlapped, and a seam calculation means for calculating a joint between frame data adjacent to each other in each overlap area of a vertical overlap area, a horizontal overlap area, and a cross overlap area; Connected frame data according to the joint calculated by the seam calculation means, and connected frames without duplication It comprises a connecting frame output means for outputting over data.

  The seam calculation means repeats the process of determining one joint from the line in the adjacent direction to the frame data in each of the vertical overlap region, the horizontal overlap region, and the cross overlap region in the direction orthogonal to the adjacent direction, Of the pixels in the direction line search range, the pixel with the smallest feature value difference from the last calculated joint pixel may be calculated as the joint in the line.

  Starting from the center of each overlap area of the vertical overlap area and the horizontal overlap area, a joint is calculated in a predetermined direction, and the position corresponding to the end joint of each overlap area adjacent in the vertical and horizontal directions of the cross overlap area is calculated. As a starting point, a process of calculating a joint of cross overlap areas and calculating a joint in each of the overlap areas of the vertical overlap area and the horizontal overlap area may be performed in parallel.

  Calculate the joint in the specified direction starting from the center of the cross overlap area, and when it reaches the end of the cross overlap area, in the other overlap areas adjacent in the predetermined direction, the position corresponding to the joint at the end of the cross overlap area is the start As described above, the process of calculating the joints and calculating the joints in the overlapping areas of the vertical overlapping area and the horizontal overlapping area may be performed in parallel.

  When there is no delay in the concatenated frame data output by the concatenated frame output means, the seam calculating means performs processing for calculating joints in parallel areas in the vertical overlap area and the horizontal overlap area in parallel, and the concatenated frame output means When there is a delay with respect to the connected frame data output by, the seam calculation means may perform the process of calculating the joints in the vertical overlap region, the horizontal overlap region, and the cross overlap region in parallel.

  For each corresponding pixel in each overlapped frame data, a connected frame output unit is provided that includes a seam correction unit that corrects a joint calculated by the seam calculation unit based on a correction element that avoids a pixel having a large luminance difference from the joint. May connect the frame data according to the joint corrected by the seam correction means, and output connected frame data without duplication.

  Seam correction means for correcting the joint calculated by the seam calculation means based on a correction element that avoids the edge portion from the joint, and the connected frame output means outputs each frame data according to the joint corrected by the seam correction means. It is also possible to output concatenated frame data that does not overlap.

  The connected frame output means comprises seam correction means for correcting the joint calculated by the seam calculation means based on a correction element for avoiding a pixel far from the joint in the frame data immediately before the frame data to be processed from the joint. The frame data may be connected in accordance with the joint corrected by the seam correction means, and connected frame data without duplication may be output.

  Based on a correction element that avoids a preset prohibited pixel from the joint, the seam calculation unit includes a seam correction unit that corrects the joint calculated by the seam calculation unit, and the connected frame output unit follows the joint corrected by the seam correction unit. Each frame data may be connected and connected frame data without duplication may be output.

  The second feature of the present invention relates to a video processing method for connecting a plurality of video data that are photographed by overlapping a part of a photographing region adjacent to each other in the vertical direction and the horizontal direction. A video processing method according to a second feature of the present invention includes a frame acquisition step of acquiring a plurality of frame data shot simultaneously from a plurality of video data, and a plurality of feature points that are coincident with each other in adjacent frame data. From the overlapped frame data that overlaps the frame data, the vertical overlap area where the frame data adjacent in the vertical direction only overlaps, the horizontal overlap area where the frame data adjacent only in the horizontal direction overlaps, and the vertical and horizontal adjacent An area setting step for setting a cross-overlapping area in which frame data to be overlapped, a seam calculation step for calculating a joint between frame data adjacent to each other in each of the vertical overlapping area, the horizontal overlapping area, and the cross-overlapping area; Connect each frame data according to the joint calculated in the seam calculation step, It comprises a connecting frame output step of outputting have connecting frame data.

  A third feature of the present invention relates to a video processing device program for causing a computer to function as the video processing device according to the first feature of the present invention.

  According to the present invention, it is possible to provide a video processing apparatus, a video processing method, and a video processing program capable of realizing high-speed processing for connecting a plurality of video data in the vertical direction and the horizontal direction.

It is a figure explaining the hardware constitutions and functional block of the video processing apparatus which concerns on embodiment of this invention. It is a flowchart explaining the process of the video processing apparatus which concerns on embodiment of this invention. It is a figure explaining the overlap frame data in the video processing device concerning an embodiment of the invention. It is a figure explaining the overlapping area | region used when the video processing apparatus which concerns on embodiment of this invention calculates a joint. It is a figure explaining the joint calculated by the video processing device concerning an embodiment of the invention. It is a figure explaining the process which the video processing apparatus which concerns on embodiment of this invention calculates the joint of each line. It is a figure explaining the process which the image processing apparatus which concerns on embodiment of this invention sets a path | route path. It is a flowchart explaining the process which calculates the joint from the periphery to a center direction in the seam calculation process by the seam calculation means which concerns on embodiment of this invention. It is a figure explaining the process which calculates a joint from the periphery to a center direction in the seam calculation process by the seam calculation means which concerns on embodiment of this invention. It is a flowchart explaining the process which calculates the joint from the center to the peripheral direction in the seam calculation process by the seam calculation unit according to the embodiment of the present invention. It is a figure explaining the process which calculates the joint from the center to the peripheral direction in the seam calculation process by the seam calculation means according to the embodiment of the present invention. It is a flowchart explaining the seam correction process by the seam correction means according to the embodiment of the present invention.

  Next, embodiments of the present invention will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

(Overview)
The video processing apparatus 1 according to the embodiment of the present invention shown in FIG. 1 connects a plurality of video data that are shot adjacent to each other in the vertical direction and the horizontal direction and overlap a part of the shooting area. The video processing device 1 searches for appropriate joints (seams) in the frame data simultaneously captured from the video data, and reconstructs the video data by connecting the frame data at the searched joints.

  Here, “adjacent to each other in the vertical direction and horizontal direction” means that certain video data is photographed by overlapping a part of the photographing area adjacent to other video data in the vertical direction, and other video data. It is meant that a part of the photographing area is overlapped and photographed adjacent to each other in the horizontal direction. Also, “adjacent to each other in the vertical and horizontal directions” means that certain video data is shot with a part of the shooting area overlapping with other video data in the vertical and horizontal directions, that is, diagonally. Including. The video processing apparatus 1 according to the embodiment of the present invention reconstructs video data obtained from a plurality of cameras arranged in vertical M rows and horizontal N columns. Note that the video data and the camera that outputs the video data do not have to be adjacent in the vertical direction or the horizontal direction in a strict sense.

  The video processing apparatus 1 shown in FIG. 1 connects a plurality of video data input from a plurality of cameras C1, C2, C3, and C4, and outputs one video data. In the embodiment of the present invention, a case will be described in which a plurality of cameras C1, C2, C3, and C4 are arranged in two vertical rows and two horizontal columns, respectively, but the present invention is not limited to this. The number of cameras that shoot video data input to the video processing device 1 is six, for example, and can be applied to any number of cameras, such as being arranged in two rows and three columns.

  With reference to FIG. 2, the video processing method by the video processing apparatus 1 according to the embodiment of the present invention will be described. In the video processing method, the processing in steps S1 to S8 is repeated for each piece of frame data captured simultaneously from video data output from each of the plurality of cameras C1, C2, C3, and C4.

  First, in step S1, the video processing apparatus 1 acquires each frame data photographed at the same time from video data output from each of the plurality of cameras C1, C2, C3, and C4.

  In step S2, the video processing apparatus 1 associates feature points with each frame data acquired in step S1, and generates overlapping frame data F0 shown in FIG.

  For example, as shown in FIG. 3, consider the upper left frame data F1, the upper right frame data F2, the lower left frame data F3, and the lower right frame data F4 that are simultaneously captured. The upper right side of the shooting area of the upper left frame data F1 overlaps the upper left side of the shooting area of the upper right frame data F2. The lower right side of the shooting area of the lower left frame data F3 and the lower left side of the shooting area of the lower right frame data F4 overlap. The lower left side of the shooting area of the upper left frame data F1 and the upper left side of the shooting area of the lower left frame data F3 overlap. The lower right side of the shooting area of the upper right frame data F2 and the upper right side of the shooting area of the lower right frame data F4 overlap. Also, the lower right side of the shooting area of the upper left frame data F1, the lower left side of the shooting area of the upper right frame data F2, the upper right side of the shooting area of the lower left frame data F3, and the upper left side of the shooting area of the lower right frame data F4. Are overlapping.

  The upper left frame data F1 and the upper right frame data F2 are such that the feature point photographed on the upper right side of the photographing area of the upper left frame data F1 and the feature point photographed on the upper left side of the photographing area of the upper right frame data F2 match. And repeat. Similarly, the lower left frame data F3 so that the feature point photographed at the lower right side of the photographing area of the lower left frame data F3 matches the feature point photographed at the lower left side of the photographing area of the lower right frame data F4. And lower right frame data F4. The upper left frame data F1 and the lower left frame data F3 are set so that the feature point photographed at the lower left side of the photographing area of the upper left frame data F1 matches the feature point photographed at the upper left side of the photographing area of the lower left frame data F3. And repeat. Upper right frame data F2 and lower right so that the feature point imaged on the lower right side of the imaging area of upper right frame data F2 matches the feature point imaged on the upper right side of the imaging area of lower right frame data F4. The frame data F4 is overlapped. Furthermore, the feature point imaged at the lower right side of the imaging area of the upper left frame data F1, the feature point imaged at the lower left side of the imaging area of the upper right frame data F2, and the upper right side of the imaging area of the lower left frame data F3. The upper left frame data F1, the upper right frame data F2, the lower left frame data F3, and the lower right frame data are set so that the photographed feature points coincide with the feature points photographed on the upper left side of the photographing area of the lower right frame data F4. Overlay with F4.

  As a result, as described in the center of FIG. 3, overlapping frame data F0 in which overlapping portions are formed in a cross shape is obtained. The frame data shown in FIG. 2 is schematically shown so that the overlapping frame data F0 is rectangular. However, when the angles of the cameras are different, some frame data are arranged obliquely. There is also.

  In step S3 of FIG. 2, the video processing device 1 sets a vertical overlap region, a horizontal overlap region, and a cross overlap region in the overlap frame data F0 generated in step S2.

  With reference to FIG. 4, each overlapping area of the overlapping frame data F0 will be described. As shown in FIG. 4, the video processing device 1 defines an overlapping area in which the upper left frame data F1 and the lower left frame data F3 overlap only in the vertical direction as a left vertical overlapping area R1. The video processing device 1 sets the position information of the left vertical overlap region R1 in the region setting data 13. The video processing device 1 defines an overlapping area where the upper right frame data F2 and the lower right frame data F4 overlap only in the vertical direction as a right vertical overlapping area R2. The video processing device 1 sets the position information of the right vertical overlap region R2 in the region setting data 13. The video processing device 1 defines an overlapping region in which the upper left frame data F1 and the upper right frame data F2 overlap only in the horizontal direction as an upper horizontal overlapping region R3. The video processing device 1 sets the position information of the upper lateral overlap region R3 in the region setting data 13. The video processing apparatus 1 defines an overlapping region where the lower left frame data F3 and the lower right frame data F4 overlap only in the horizontal direction as a lower horizontal overlapping region R4. The video processing device 1 sets the position information of the lower horizontal overlap region R4 in the region setting data 13. The video processing apparatus 1 defines an overlapping region in which the upper left frame data F1, the upper right frame data F2, the lower left frame data F3, and the lower right frame data F4 overlap in the vertical direction and the horizontal direction as a cross overlapping region R5. The video processing device 1 sets the position information of the cross overlap region R5 in the region setting data 13.

  When N cameras are arranged vertically and M cameras are arranged horizontally, the number of vertically overlapping areas is M (N−1), and the number of horizontally overlapping areas is N (M -1). The cross overlap area is (N + 1) (M + 1) -2 (M + N).

  In step S4 of FIG. 2, the video processing apparatus 1 calculates the horizontal joint SEW and the vertical joint SEL shown in FIG. The horizontal seam SEW and the vertical seam SEL are seams when creating frame data by connecting the frame data.

  If it is determined in step S5 that there is no seam correction instruction, the process proceeds to step S7. On the other hand, when it is determined in step S5 that there is a seam correction instruction, in step S6, the horizontal seam SEW and vertical seam SEL calculated in step S4 are corrected, and the process proceeds to step S7.

  In step S7, the video processing apparatus 1 connects the frame data according to the joints. Specifically, the video processing apparatus 1 cuts the upper left frame data F1, the upper right frame data F2, the lower left frame data F3, and the lower right frame data F4 at the horizontal joint SEW and the vertical joint SEL, respectively, to thereby obtain the upper left frame data f1 and the upper right frame. Data f2, lower left frame data f3, and lower right frame data f4 are generated. The video processing apparatus 1 connects the upper left frame data f1, the upper right frame data f2, the lower left frame data f3, and the lower right frame data f4 to generate the concatenated frame data 16.

  In step S8, the video processing apparatus 1 outputs the concatenated frame data 16 generated in step S7. The concatenated frame data 16 is frame data of video data output from the video processing device 1.

(Video processing device)
A video processing apparatus 1 shown in FIG. 1 is a general computer including a storage device 10, a processing device 20, and an input / output interface 30. The functions shown in FIG. 1 are realized by a general computer executing a video processing program.

  The storage device 10 is a ROM (Read Only Memory), a RAM (Random access memory), a hard disk or the like, and stores various data such as input data, output data, and intermediate data for the processing device 20 to execute processing. . The processing device 20 is a CPU (Central Processing Unit), and reads / writes data stored in the storage device 10 and inputs / outputs data to / from the input / output interface 30 to execute processing in the video processing device 1. To do.

  The input / output interface 30 receives input from the input device I such as a keyboard and a mouse, and a plurality of cameras C 1, C 2, C 3 and C 4, and inputs the input data to the processing device 20. The input / output interface 30 further outputs the processing result by the processing device 20 to an output device O such as a display. The input device I, the plurality of cameras C1, C2, C3, C4, and the output device O may be connected to the video processing device 1 via a communication interface and a communication network. Alternatively, instead of the plurality of cameras C1, C2, C3 and C4, a video recorder or storage device storing a plurality of pre-captured shooting data may be connected to the video processing device 1 to process the video processing data. good.

  The storage device 10 stores a video processing program and stores setting data 11, frame information data 12, region setting data 13, seam data 14, correction seam data 15, and linked frame data 16.

  The setting data 11 stores parameters necessary for processing by the video processing apparatus 1 according to the embodiment of the present invention. The setting data 11 includes, for example, parameters such as the number and arrangement of video data input to the video processing device 1 and whether to calculate a high-quality joint.

  The frame information data 12 is information of each frame data photographed simultaneously in a plurality of video data output from the plurality of cameras C1, C2, C3, and C4. The frame information data 12 is data in which information such as a pixel value, a frame rate, and a luminance value is associated with a camera identifier.

  The area setting data 13 is data defining position information of each overlapping area when a vertical overlapping area, a horizontal overlapping area, and a cross overlapping area are set according to the overlapping state of each frame data in the overlapping frame data. In the case of the example shown in FIG. 4, the area setting data 13 includes one of the overlapping areas of the left vertical overlapping area R1, the right vertical overlapping area R2, the upper horizontal overlapping area R3, the lower horizontal overlapping area R4, and the cross overlapping area R5. Define.

  The seam data 14 is data obtained as a result of calculating the joint of each frame data based on the area setting data 13 and the frame information data 12.

  The corrected seam data 15 is obtained by correcting the seam data 14 from various viewpoints. The corrected seam data 15 is used to obtain video data with higher accuracy than the seam data 14.

  The concatenated frame data 16 is generated by connecting a plurality of frame data captured simultaneously from each of a plurality of video data output from the plurality of cameras C1, C2, C3, and C4 according to the seam data 14 or the corrected seam data 15. The The concatenated frame data 16 becomes one frame data constituting the video data output from the video processing device 1.

  The processing device 20 includes setting acquisition means 21, frame acquisition means 22, area setting means 23, seam calculation means 24, seam correction means 25, and connected frame output means 26.

(Setting acquisition means)
The setting acquisition unit 21 acquires parameters necessary for processing by the video processing apparatus 1 according to the embodiment of the present invention, and stores setting data 11 including the acquired parameters. The setting acquisition unit 21 acquires parameters according to information input from the input device I by the user. Alternatively, the setting acquisition unit 21 acquires parameters necessary for processing by the video processing apparatus 1 by analyzing each video data input from the plurality of cameras C1, C2, C3, and C4.

(Frame acquisition means)
The frame acquisition means 22 acquires a plurality of frame data photographed simultaneously from the video data input from the plurality of cameras C1, C2, C3, and C4. The frame acquisition unit 22 generates and stores frame information data 12 for each of a plurality of acquired frame data.

(Area setting means)
The region setting unit 23 matches the feature points of the frame data adjacent to each other in the plurality of frame data photographed at the same time, and generates overlapping frame data by superimposing the plurality of frame data photographed at the same time. In the example shown in FIG. 3 and FIG. 4, the area setting means 23 is based on the generated overlapping frame data, and the vertical overlapping areas R1 and R2 in which the frame data adjacent only in the vertical direction overlap, and the frames adjacent only in the horizontal direction. The horizontal overlap areas R3 and R4 where data overlap and the cross overlap area R5 where frame data adjacent in the vertical and horizontal directions overlap are set. The area setting unit 23 stores the position information of each overlapping area in the area setting data 13.

(Seam calculation means)
The seam calculation unit 24 repeats the process of determining one joint from the line in the adjacent direction to the frame data in each overlapping area of the vertical overlapping area, the horizontal overlapping area, and the cross overlapping area in a direction orthogonal to the adjacent direction. The seam calculation means 24 searches for a frame data joint in the horizontal direction in the vertical overlap region, searches for a frame data joint in the vertical direction in the horizontal overlap region, and frames in the vertical and horizontal directions in the cross overlap region. Search for data joints. The seam calculation unit 24 calculates, as a joint in this line, a pixel having the smallest difference in feature value from the pixel of the joint calculated immediately before among the pixels in the search range of the line in the adjacent direction.

  As shown in FIG. 4, the seam calculation means 24 has a horizontal seam SEW and a vertical seam for smoothly connecting the upper left frame data F1, the upper right frame data F2, the lower left frame data F3, and the lower right frame data F4. SEL is calculated. The horizontal joint SEW is provided in a substantially horizontal direction (left and right direction of the paper surface shown in FIG. 4) in the left vertical overlap region R1, the cross overlap region R5, and the right vertical overlap region R2. The vertical joints SEL are provided in a substantially vertical direction (up and down direction on the paper surface shown in FIG. 4) in the upper lateral overlap region R3, the cross overlap region R5, and the lower lateral overlap region R4.

(Seam calculation process)
The seam calculation unit 24 determines one joint from the lines in the adjacent direction of the frame data, and sequentially determines joints in a direction orthogonal to the adjacent direction. For example, since the frame data is adjacent to the left vertical overlap region R1 in the vertical direction, the seam calculation unit 24 determines one pixel as a joint from the vertical lines of the left vertical overlap region R1. Further, the seam calculation means 24 sequentially calculates the joints in the horizontal direction, and identifies the pixels that form the horizontal joint SEW. Similarly, since the frame data is adjacent in the horizontal direction in the upper horizontal overlap region R3, the seam calculation unit 24 determines one pixel from the horizontal line of the upper horizontal overlap region R3 as a joint. Further, the seam calculation means 24 sequentially calculates the joints in the vertical direction, and identifies the pixels forming the vertical joint SEL. Further, in the cross overlap region R5, a pixel that forms a horizontal seam SEW is specified by searching for a horizontal seam, and a pixel that forms a vertical seam SEL is specified by searching for a seam in the vertical direction.

  With reference to FIG. 6 and FIG. 7, the process by which the seam calculation means 24 specifies the joint will be described. The example shown in FIGS. 6 and 7 searches for a joint from the lower line to the upper line in the horizontal overlap region where the frame data overlaps in the horizontal direction.

  The seam calculation unit 24 determines a search range in the horizontal direction in the horizontal overlap region. This search range is composed of an odd number of pixels such as 5 pixels and 7 pixels. By limiting the search range of the joint, it can be expected to speed up the processing. As another example, all the pixels in one horizontal line of the horizontal overlap region may be set as the search range.

  The seam calculation unit 24 calculates the accumulated cost for each pixel in the search range, and determines the pixel with the lowest accumulated cost as the joint pixel in the search range. Specifically, in the search range of the previous line shown in FIG. 6, the pixel with the lowest accumulated cost is recorded as the optimum path at the joint of the previous line. When searching for a joint on the target line, among the pixels in the search range of the target line, the pixel having the closest feature to the joint of the previous line is recorded as the optimum path of the joint of the target line. The

  Here, the accumulated cost Cp (n) of the target pixel p (n) in the target line will be described. The accumulated cost Cp (n) of the target pixel p (n) is determined immediately before the difference D between the feature values of the joint pixel P (n−1) and the target pixel p (n) determined immediately before. This is the sum of the accumulated cost CP (n−1) at the joint pixel. Here, the feature value is calculated based on the feature amount of the pixel, and may be a color difference, a luminance difference, or a value obtained by mixing the color difference and the luminance difference, for example.

  When the pixels constituting the joint are specified in each line, the joint is constructed by tracing the pixels constituting the joint as shown in FIG. As shown in FIG. 6, when a joint having a minimum accumulated cost is specified from the bottom to the top, the joint may be configured from the top to the bottom as shown in FIG. 7.

  In order to further simplify the processing, a difference between feature values is calculated for each pixel constituting the search range of the line of interest and the joint pixel specified in the previous line, and the pixels belonging to the search range with the smallest difference May be identified as a pixel at the joint of the line. Further, each time a joint is specified in each line, it may be connected to the joint specified immediately before to form a joint. The process when searching for a joint in the vertical overlap region is the same.

  With reference to FIGS. 8 to 11, a processing order for realizing high-speed processing and video quality improvement in the embodiment of the present invention will be described.

(When processing from the periphery to the center)
First, with reference to FIG. 8 and FIG. 9, processing for calculating a joint in an overlapping region other than the cross overlapping region and then calculating a joint in the cross overlapping region will be described. The seam calculation means 24 calculates a joint in a predetermined direction starting from the center of each overlap region of the vertical overlap region and the horizontal overlap region, and ends the overlap regions adjacent to each other in the vertical direction and the horizontal direction of the cross overlap region. Starting from the position corresponding to the joint, the joint of the cross overlap area is calculated. The seam calculation unit 24 performs processing for calculating the joints in parallel in each of the vertical overlap area and the horizontal overlap area.

  As shown in FIG. 8, the seam calculation unit 24 searches for pixels constituting the joint by calculating the accumulated cost in the horizontal direction starting from the center of the vertical overlap region in step S <b> 101. The process in step S101 is repeated until the end of the vertical overlap region is reached, as shown in step S102.

  In parallel with the processing of step S101 and step S102, the seam calculation means 24 searches the pixels constituting the joint by calculating the accumulated cost in the vertical direction starting from the center of the horizontal overlap region in step S103. The process in step S103 is repeated until the end of the horizontal overlap region is reached, as shown in step S104.

  When the joint is searched to the end in each of the vertical overlap region and the horizontal overlap region, more specifically, when the joint search process in the vertical overlap region and the horizontal overlap region reaches the boundary with the cross overlap region, The process proceeds to S105.

  In step S105, the seam calculation unit 24 sets the accumulated cost of each end portion in contact with the cross overlap region of the vertical overlap region and the horizontal overlap region to the accumulated cost of each end portion of the cross overlap region. The seam calculation unit 24 searches for a joint toward the center of the cross overlap area based on the accumulated cost set from the calculation results of the vertical overlap area and the horizontal overlap area.

  In step S105, the accumulated cost is calculated in the center direction of the cross-overlapping region, and when the joint is determined for each line, in step S106, the pixel having the smallest cost calculated in each line is selected to determine the joint.

  Specifically, as shown in (1) of FIG. 9, the seam calculation unit 24 searches for a joint in the left-right direction starting from the respective centers of the left vertical overlap region R1 and the right vertical overlap region R2. In parallel with this, the seam calculation means 24 searches for a joint in the vertical direction starting from the center of each of the upper lateral overlap region R3 and the lower lateral overlap region R4.

  When a search is made up to the end in contact with the cross overlap region R5 in the left and right overlap regions, a joint is searched in the cross overlap region R5 as shown in (2) of FIG. When the joint is searched to the right end in the left vertical overlap region R1, the cumulative cost at the right end of the left vertical overlap region R1 is set to the cumulative cost at the left end of the cross overlap region R5, and the joint is searched in the center direction. . When the joint is searched to the left end in the right vertical overlap region R2, the joint cost at the left end of the right vertical overlap region R2 is set to the cumulative cost at the right end of the cross overlap region R5, and the joint is searched in the center direction. . When the joint is searched to the lower end in the upper lateral overlap region R3, the joint cost at the lower end of the upper lateral overlap region R3 is set to the cumulative cost at the upper end of the cross overlap region R5, and the joint is searched in the center direction. . When the joint is searched to the upper end in the lower lateral overlap region R4, the cumulative cost at the upper end of the lower lateral overlap region R4 is set to the cumulative cost at the lower end of the cross overlap region R5, and the joint is formed in the center direction. Explore.

  According to the example shown in FIGS. 8 and 9, the search for the joint can be processed in parallel in each of the overlapping regions of the vertical overlapping region and the horizontal overlapping region, so that high-speed processing can be expected. Also, in the cross overlap area, the calculation results of the vertical overlap area and the horizontal overlap area are limited, so it is smoother considering the relation with other areas compared to searching for joints in the cross overlap area alone. It becomes possible to set a joint.

(When processing from the center to the periphery)
Next, with reference to FIG. 10 and FIG. 11, a process for calculating a joint in a cross overlap area and then calculating a joint in an overlap area other than the cross overlap area will be described. The seam calculation means 24 calculates a joint in a predetermined direction starting from the center of the cross overlap area, and when it reaches the end of the cross overlap area, in another overlap area adjacent in the predetermined direction, the seam at the end of the cross overlap area Each joint is calculated starting from the position corresponding to. The seam calculation unit 24 performs processing for calculating the joints in parallel in each of the vertical overlap area and the horizontal overlap area.

  As shown in FIG. 9, the seam calculation means 24 searches for pixels constituting the joint by calculating the accumulated cost in the vertical and horizontal directions starting from the center of the cross overlap region in step S201. The process in step S201 is repeated until the end of the cross overlap area is reached, as shown in step S202.

  When the joint is searched to the end in the cross overlap region, more specifically, when the search process for the joint that can be placed in the cross overlap region reaches the boundary with the vertical overlap region, the process proceeds to step S203.

  In step S203, the seam calculation unit 24 sets the accumulated cost of each end portion in contact with the vertical overlap region of the cross overlap region to the accumulated cost of each end portion of the vertical overlap region. The seam calculation unit 24 searches for a joint in the left and right peripheral directions based on the accumulated cost set from the calculation result of the cross overlap area. The process of step S203 is repeated until the end of the vertical overlap region is reached, as shown in step S204.

  In parallel with the processing of step S203 and step S204, the seam calculation means 24 calculates the accumulated cost of each end in contact with the horizontal overlap area of the cross overlap area in step S205 as the accumulated cost of each end of the horizontal overlap area. Set to. The seam calculation unit 24 searches for a joint in the upper and lower peripheral directions based on the accumulated cost set from the calculation result of the cross overlap area. The process in step S205 is repeated until the end of the lateral overlap region is reached, as shown in step S206.

  In step S105, when the accumulated cost is calculated up to the end of the cross overlap area, in step S106, a pixel having the smallest cost calculated in each line is selected to determine a joint.

  Specifically, as shown in (1) of FIG. 11, the seam calculation unit 24 searches for a joint in the vertical and horizontal directions starting from the center of the cross overlap region R5.

  In the cross overlap area R5, when the search is made up to the ends contacting the left and right and upper and lower overlap areas, as shown in (2) of FIG. 11, a joint is searched in the left and right and upper and lower overlap areas. When the joint is searched to the left end in the cross overlap region R5, the joint cost at the left end of the cross overlap region R5 is set to the cumulative cost at the right end of the left vertical overlap region R1, and the joint is searched in the left direction. When the joint is searched to the right end in the cross overlap region R5, the joint cost at the right end of the cross overlap region R5 is set to the cumulative cost at the left end of the right vertical overlap region R2, and the joint is searched in the right direction. When the joint is searched to the upper end in the cross overlap region R5, the joint cost at the upper end of the cross overlap region R5 is set to the cumulative cost at the lower end of the upper lateral overlap region R3, and the joint is searched upward. When the joint is searched to the lower end in the cross overlap area R5, the cumulative cost of the lower end of the cross overlap area R5 is set to the accumulated cost of the upper end of the lower horizontal overlap area R4, and the joint is searched downward. .

  According to the example shown in FIG. 10 and FIG. 11, the joint search can be processed in parallel in each of the vertical overlap region and the horizontal overlap region, so that the processing speed can be expected to be increased. In addition, in each overlap area of the vertical overlap area and the horizontal overlap area, the calculation result of the cross overlap area is limited, so compared with the case of searching for a joint in the vertical overlap area and the horizontal overlap area alone, It becomes possible to set a smooth joint in consideration of the relationship.

(When processing in parallel)
In the case of “processing from the periphery to the center” and “processing from the center to the periphery”, the case has been described in which the information of the joints already processed is propagated and referred to in subsequent processing. Since such processing restricts the processing order in each overlapping area, if the processing amount in the video processing device 1 increases in a live distribution system, frame loss occurs in the video data output from the video processing device 1, and the image quality is reduced. May be disturbed.

  For example, consider a case where still images of a landscape are captured and connected by a plurality of cameras. When a gust of wind blows in this landscape and many objects such as dead leaves and cherry blossom petals begin to fly at high speed in the video, the processing amount of the video processing device 1 suddenly increases. Therefore, the processing of the video processing device 1 may not catch up with the frame rate of the original video and may cause a delay.

  Therefore, when there is no delay in the concatenated frame data output from the concatenated frame output unit 26, the seam calculation unit 24 performs a process of calculating a joint in each overlap region of the vertical overlap region and the horizontal overlap region in parallel. Here, the case where there is no delay is a case where the frequency of the video data constituted by the concatenated frame data output from the concatenated frame output unit 26 is the same as the frequency of the video data acquired by the frame acquisition unit 22.

  When the load on the video processing device 1 is not high and the video can be output at the same frequency as the input video, only the overlapping areas of the vertical overlap area and the horizontal overlap area are processed in parallel, and before and after that, A joint is calculated in the cross overlap region, and the calculation result of the preceding process is reflected in the calculation result of the subsequent process. Specifically, in the case of processing from the periphery toward the center, the seam calculation unit 24 performs joint calculation processing in the vertical overlap region and the horizontal overlap region in parallel, and calculates the end portions of the vertical overlap region and the horizontal overlap region. The joint information is propagated to the cross overlap region. In the case of processing from the center toward the periphery, a joint calculation process is performed in the cross overlap area, and the joint information at the end of the cross overlap area is propagated to the vertical overlap area and the horizontal overlap area. Thereby, when the load in the video processing device 1 is low, the quality of the video can be improved.

  On the other hand, when there is a delay with respect to the concatenated frame data output from the concatenated frame output unit 26, the seam calculation unit 24 performs processing for calculating joints in parallel in the vertical overlap region, the horizontal overlap region, and the cross overlap region in parallel. Do. Here, the case where there is a delay is a case where the frequency of the video data constituted by the concatenated frame data output from the concatenated frame output unit 26 is lower than the frequency of the original video data acquired by the frame acquisition unit 22. It is.

  When the load on the video processing device 1 is high and the video is output at a frequency lower than the frequency of the input video, the seam calculation unit 24 performs the following processing on each overlapping region of the vertical overlapping region, the horizontal overlapping region, and the cross overlapping region. Performs processing to calculate joints all at once and does not propagate processing results to other overlapping areas. Since the joints are not propagated to other overlapping regions, the smoothness of the joints may be inferior, but the processing load on the video processing device 1 can be reduced, so that video data can be output without delay. Become.

  In addition, since the load in the video processing apparatus 1 changes sequentially, the seam calculation unit 24 transmits or does not propagate the joint information to other overlapping areas according to the load on the video processing apparatus 1. It is also possible to adjust the load of the processing apparatus 1. If it is known that the video processing apparatus 1 is video data whose processing amount does not increase suddenly, the information is set in the setting data 11 so that the processing of the seam calculation means 24 is performed according to the video processing load. The change may not be applied.

(Seam correction means)
The seam correction unit 25 corrects the joint calculated by the seam calculation unit 24 to enable higher-definition video output. The seam correction means 25 corrects the joints based on several correction elements. In the embodiment of the present invention, the following four correction elements correspond. The seam correction means 25 may correct the joints using all of the four correction elements, may correct the joints using any correction element, and further, other seams shown below. You may correct a joint using an element.

  In the embodiment of the present invention, each correction element takes a value from 0 to 1 and is designed so that the closer to 1, the less likely it is to be selected as a joint.

(1) First correction element that avoids a pixel having a large luminance difference from a joint for each corresponding pixel in each overlapped frame data The first correction element is, for example, a combination target in the left vertical overlap region R1 The upper left frame data F1 and the lower left frame data F3 are correction elements for setting a joint while avoiding a pixel pair whose luminance differs greatly between corresponding pixel pairs. The first correction element focuses on the luminance difference between the corresponding pixels of the frame data that are overlapped in the overlapping regions of the vertical overlapping region, the horizontal overlapping region, and the cross overlapping region.

  The seam correction means 25 applies a low-pass filter to each pixel pair to be connected, and calculates the absolute value of the luminance difference of the pixel pair obtained as a result. The seam correction unit 25 accumulates the absolute value of the luminance difference, normalizes the result so that the maximum is 1.0, and calculates a first correction element for each pixel pair.

(2) Second correction element for avoiding the edge portion from the joint The second correction element is a correction element for setting the joint by avoiding the edge area when the edge area is included in the image. is there. The second correction element focuses on an edge portion having a large color difference of each frame data that is overlapped in each overlap region of the vertical overlap region, the horizontal overlap region, and the cross overlap region.

  The seam correction means 25 applies an edge detection filter such as a Scherr filter to the image in the horizontal and vertical directions, and calculates the L1 norm of the corresponding pixel pair as a result. The seam correction unit 25 accumulates the L1 norm calculated for each pixel pair, normalizes the maximum result to be 1.0, and calculates a second correction element for each pixel pair.

(3) Third correction element for avoiding pixels far from the joint in the frame data immediately before the frame data to be processed from the joint. The third correction element is the joint obtained by the seam calculation means 24. A low-pass filter is applied in the direction, and is a correction element for suppressing a change in joints over time and changing smoothly.

  When the image generated by the joint obtained by the seam calculation unit 24 is W and the position of the joint in an arbitrary line is Seam (y), the seam correction unit 25 is more than an arbitrary distance Ws from Seam (y). The third correction element of the separated pixels is set to 1.0, the third correction element of Seam (y) is set to 0.0, and weighting is performed so that Ws becomes 1.0. Further, the third correction element Wlpf is calculated by applying a low-pass filter so that the image W generated by the joint obtained by the seam calculation unit 24 is expressed by the equation (1) in time series.

Wlpf (n) = W (n-1) × a + Wlpf × (1-a) ・ ・ ・ ・ Equation (1)
Wlpf (n): third correction element in the frame data to be calculated for the joint
W: an image generated by the joint obtained by the seam calculation means 24
Wlpf: Third correction element in the immediately preceding frame data
a: Arbitrary coefficient

(4) Fourth correction element for avoiding preset prohibited area from joints The fourth correction element sets a conspicuous place in the frame data as a prohibited area in advance, and this prohibited area is not selected as a joint. It is a correction element to do.

  The seam correction means 25 sets the fourth correction element of the pixels in the prohibited area to 1.0, sets the fourth correction element of the pixels other than the prohibited area to 0.0, and sets the fourth correction element for each pixel to the fourth correction element. Set the correction factor. The prohibited area can be set for any pixel.

  With reference to FIG. 12, the seam correction process by the seam correction means 25 will be described.

  First, in step S301, the seam correction unit 25 calculates, for each pixel, a first correction element that avoids a pixel having a large luminance difference. In step S302, the seam correction unit 25 calculates a second correction element that avoids the edge portion for each pixel. In step S303, the seam correction unit 25 calculates, for each pixel, a third correction element that avoids a pixel far from the joint of the previous frame data. In step S304, the seam correction unit 25 calculates a fourth correction element for avoiding a preset prohibited area for each pixel.

  In step S305, the seam correction unit 25 corrects the joint using any one or more of the first to fourth correction elements calculated in steps S301 to S304.

  In step S301 to step S304, each correction element is calculated for each pixel constituting the search target of each line. As another example, the calculation may be performed for all pixels constituting each line in each overlapping region. Note that correction elements that are not referred to in step S305 need not be calculated.

(Connected frame output means)
The connected frame output means 26 connects the respective frame data according to the joint calculated by the seam calculating means 24 or the joint corrected by the seam correcting means 25, and outputs the connected frame data 16 having no overlap. The concatenated frame output means 26 intermittently outputs the concatenated frame data 16 and outputs it to the output device O as video data.

  As described above, the video processing apparatus 1 according to the embodiment of the present invention can improve the image quality by appropriately calculating the joint of the frame data pair to be joined at the same time while realizing high-speed processing. Since it is possible, a “fast and beautiful composite video” can be generated.

(Other embodiments)
As described above, the embodiments of the present invention have been described. However, it should not be understood that the descriptions and drawings constituting a part of this disclosure limit the present invention. From this disclosure, various alternative embodiments, examples, and operational techniques will be apparent to those skilled in the art.

  For example, the video processing apparatus described in the embodiment of the present invention may be configured on one piece of hardware as shown in FIG. 1, or may be configured on a plurality of pieces of hardware according to the functions and the number of processes. May be. Further, it may be realized on an existing video processing system.

  In addition, as proposed in Non-Patent Document 2, the video processing method according to the embodiment of the present invention calculates the joint of each frame data using the reduced frame data and outputs the concatenated frame data. good. This can be expected to speed up the processing.

  It goes without saying that the present invention includes various embodiments not described herein. Therefore, the technical scope of the present invention is defined only by the invention specifying matters according to the scope of claims reasonable from the above description.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Storage apparatus 11 Setting data 12 Frame information data 13 Area setting data 14 Seam data 15 Correction seam data 16 Concatenated frame data 20 Processing apparatus 21 Setting acquisition means 22 Frame acquisition means 23 Area setting means 24 Seam calculation means 25 Seam Correction means 26 Connection frame output means 30 Input / output interface C1, C2, C3, C4 Camera I Input device O Output device

Claims (11)

A video processing device that connects a plurality of video data that are photographed by overlapping a part of a photographing region adjacent to each other in a vertical direction and a horizontal direction,
Frame acquisition means for acquiring a plurality of frame data captured simultaneously from the plurality of video data;
Frame data adjacent to each other only in the horizontal direction from the overlapping frame data obtained by matching the feature points of the adjacent frame data and overlapping the plurality of frame data, and overlapping in the frame data adjacent in the vertical direction only An area setting means for setting a horizontal overlap area in which overlaps with each other and a cross overlap area in which frame data adjacent in the vertical direction and the horizontal direction overlap,
Seam calculation means for calculating a joint between frame data adjacent to each other in each overlap region of the vertical overlap region, the horizontal overlap region, and the cross overlap region;
A video processing apparatus comprising: connected frame output means for connecting the respective frame data according to the joint calculated by the seam calculating means and outputting connected frame data having no overlap. The seam calculating means includes
In each overlap region of the vertical overlap region, the horizontal overlap region and the cross overlap region, the process of determining one joint from the line in the adjacent direction with the frame data is repeated in a direction orthogonal to the adjacent direction,
The pixel having the smallest feature value difference from the pixel of the joint calculated immediately before among the pixels in the search range of the line in the adjacent direction is calculated as the joint in the line. The video processing apparatus described. Starting from the center of each overlap region of the vertical overlap region and the horizontal overlap region, the joint is calculated in a predetermined direction, and the end joint of each overlap region adjacent to the cross overlap region in the vertical direction and the horizontal direction, respectively. Starting from the position corresponding to, the joint of the cross overlap region is calculated,
The video processing apparatus according to claim 2, wherein the process of calculating the joint is performed in parallel in each of the overlapping areas of the vertical overlapping area and the horizontal overlapping area. Starting from the center of the cross overlap area, the joint is calculated in a predetermined direction, and when reaching the end of the cross overlap area, in another overlap area adjacent to the predetermined direction, the joint at the end of the cross overlap area Calculate the joints starting from the corresponding positions,
The video processing apparatus according to claim 2, wherein the process of calculating the joint is performed in parallel in each of the overlapping areas of the vertical overlapping area and the horizontal overlapping area. When there is no delay for the concatenated frame data output by the concatenated frame output means,
The seam calculation means performs the process of calculating the joint in parallel in each of the vertical overlap area and the horizontal overlap area,
When there is a delay for the concatenated frame data output by the concatenated frame output means,
5. The video processing apparatus according to claim 3, wherein the seam calculation unit performs a process of calculating joints in the vertical overlap region, the horizontal overlap region, and the cross overlap region in parallel. Seam correction means for correcting the joint calculated by the seam calculation means based on a correction element for avoiding pixels having a large luminance difference from the joint for each corresponding pixel in each overlapping frame data,
The video processing apparatus according to claim 1, wherein the connected frame output unit connects the frame data according to the joint corrected by the seam correction unit, and outputs connected frame data having no overlap. Seam correction means for correcting the joint calculated by the seam calculation means based on a correction element for avoiding the edge portion from the joint,
The video processing apparatus according to claim 1, wherein the connected frame output unit connects the frame data according to the joint corrected by the seam correction unit, and outputs connected frame data having no overlap. A seam correction unit that corrects the joint calculated by the seam calculation unit based on a correction element that avoids a pixel far from the joint in the frame data immediately before the frame data to be processed from the joint,
The video processing apparatus according to claim 1, wherein the connected frame output unit connects the frame data according to the joint corrected by the seam correction unit, and outputs connected frame data having no overlap. A seam correction unit that corrects a joint calculated by the seam calculation unit based on a correction element that avoids a preset prohibited pixel from the joint;
The video processing apparatus according to claim 1, wherein the connected frame output unit connects the frame data according to the joint corrected by the seam correction unit, and outputs connected frame data having no overlap. A video processing method for connecting a plurality of video data shot by overlapping a part of a shooting area adjacent to each other in a vertical direction and a horizontal direction,
A frame acquisition step of acquiring a plurality of frame data shot simultaneously from the plurality of video data;
Frame data adjacent to each other only in the horizontal direction from the overlapping frame data obtained by matching the feature points of the adjacent frame data and overlapping the plurality of frame data, and overlapping in the frame data adjacent in the vertical direction only A region setting step for setting a horizontal overlap region where the frame data overlaps and a cross overlap region where the frame data adjacent in the vertical direction and the horizontal direction overlap,
A seam calculating step for calculating a joint between adjacent frame data in each of the overlapping regions of the vertical overlapping region, the horizontal overlapping region, and the cross overlapping region;
A video processing method comprising: a connected frame output step of connecting each frame data according to the joint calculated in the seam calculating step and outputting connected frame data without duplication.   A video processing program for causing a computer to function as the video processing device according to any one of claims 1 to 9.

Images