JP5421939B2 - Color correction apparatus, color correction method, and color correction program - Google Patents

Description

  The present invention relates to a technique for correcting the color of a moving image.

  When generating stereoscopic images that can be stereoscopically viewed using parallax, a pair of photographing means arranged at different positions, for example, a compound eye photographing device such as a stereo camera is used. In general, cameras of the same model can easily adjust the color tone of an image acquired only by setting the camera, so that it is easy to reduce the color difference between images.

  Conventional techniques for performing color correction in order to reduce the color difference between two cameras of different models include a method of performing color correction using a conversion polynomial (see Non-Patent Document 1) and image histograms. Thus, a method of performing color correction (see Non-Patent Document 2) is disclosed.

Inamura, Taguchi, “Color adjustment between different cameras”, IEICE Technical Report SIS, March 2007, Vol. 106, No. 575, p. 19-22 Inamura, Taguchi, “Color adjustment method between different cameras”, IEICE Technical Report, December 2007, Vol. 107, No. 374, p. 13-18

  However, in Non-Patent Document 1, it is necessary to include a color chart in the subject. In Non-Patent Document 2, when the two cameras are separated or the angle is large, for example, in a scene where the target color is visible in one image but not the other image due to the angle problem. There was a problem that color correction was not supported.

  The present invention has been made in view of the above, and an object of the present invention is to reduce a color difference generated between two moving images obtained when two cameras with different specifications are photographed side by side with the same subject.

According to a first aspect of the present invention, there is provided a color correction apparatus comprising: an input unit that inputs two moving images obtained by photographing the same subject and stores them in a storage unit; and the two moving images stored in the storage unit. Extraction means for extracting a frame image at a time, area division means for classifying each pixel of each frame image into one of a plurality of areas, a corresponding area between the frame images is determined, and a color for each area Region determining means for obtaining a correspondence relationship between the color image, learning means for learning a conversion rule to be the other color when one color is converted based on the color correspondence relationship, and using the conversion rule, And color correction means for correcting one of the colors.

  In the color correction apparatus, the area determination unit sets a scanning area for each frame image and scans each frame image, and the area existing in the scanning area for each scanning area and the color of the area And the corresponding region is determined by comparing the color of the region existing in the scanning region between the corresponding scanning regions of each of the frame images.

  In the color correction apparatus, the area determination unit creates a scan result table in which the area existing in the scan area and the color of the area are described for each scan area, and the scan area of the scan result table is displayed. A search is performed to obtain the Euclidean distance of the color of the region between the corresponding scanning regions in order from the scanning region in which the number of the regions existing in the corresponding scanning region of each of the frame images is small. The region having the closest Euclidean distance is determined as the corresponding region.

  In the color correction apparatus, a frame image at the same time is extracted from each of the moving image corrected by the color correcting unit and the other moving image to obtain a color correspondence relationship, and a color difference between corresponding colors exceeds a predetermined threshold value. In such a case, it is characterized by having determination means for performing additional learning.

In the color correction method according to the second aspect of the present invention, the step of inputting two moving images obtained by photographing the same subject and storing them in the storage means and the two moving images stored in the storage means at the same time Extracting a frame image in each of the above, a step of classifying each pixel of each of the frame images into one of a plurality of regions , a step of determining a corresponding region between the frame images, and a color for each corresponding region A step of obtaining a correspondence relationship, a step of learning a conversion rule that becomes the other color when one color is converted based on the correspondence relationship of the color, and one color of the moving image using the conversion rule And a step of correcting.

  In the color correction method, the step of determining the corresponding area sets a scanning area for each of the frame images, scans each of the frame images, and each of the scanning areas includes the area existing in the scanning area. The color of the region is obtained, and the corresponding region is determined by comparing the color of the region existing in the scanning region between the corresponding scanning regions of each of the frame images.

  In the color correction method, the step of determining the corresponding region creates a scanning result table in which the region existing in the scanning region and the color of the region are described for each scanning region, The scanning area is searched, and the Euclidean distance of the color of the area is determined between the corresponding scanning areas in order from the scanning area where the number of the areas existing in the corresponding scanning area of each of the frame images is small, The region having the Euclidean distance that is equal to or smaller than a predetermined threshold is determined as a corresponding region.

  In the above color correction method, a frame image at the same time is extracted from each of the corrected moving image and the other moving image to obtain a color correspondence relationship, and if the color difference between the corresponding colors exceeds a predetermined threshold value, add It has the step which performs learning, It is characterized by the above-mentioned.

  A color correction program according to a third aspect of the present invention causes a computer to execute the color correction method.

  According to the present invention, it is possible to reduce the color difference between two moving images obtained when two cameras with different specifications are photographed side by side with the same subject.

It is a block diagram which shows the structure of the color correction apparatus in this Embodiment. 6 is a flowchart illustrating a flow of processing in which the color correction apparatus according to the present embodiment reduces a color difference between moving images. It is a figure which shows a mode that the frame image was extracted from each of two moving images. It is a figure which shows the example which produced | generated the area | region image from the frame image. It is a figure which shows the example of a correspondence table. It is a figure which shows the example of learning using a neural network. It is a flowchart which shows the flow of the process which scans an area | region image. It is a figure which shows the example of the window for image scanning. It is a figure which shows a mode that the whole area | region image is scanned using a window. It is a figure which shows the example of a scanning result table | surface. It is a flowchart which shows the flow of the process which produces | generates a correspondence from a scanning result table | surface. It is explanatory drawing of the process which determines a corresponding area | region using a Euclidean distance. It is a figure which shows the example of the scanning result table | surface which deleted the determined area | region. It is a block diagram which shows the structure of another color correction apparatus in this Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing the configuration of the color correction apparatus according to this embodiment. The color correction apparatus 1 shown in the figure includes a frame extraction unit 2, a region division unit 3, a region comparison unit 4, a corresponding color learning unit 5, a color correction moving image generation unit 6, a color difference evaluation unit 7, a media control unit 8, and A display unit 9 is provided. Each unit included in the color correction apparatus 1 may be configured by a computer including an arithmetic processing device, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the color correction apparatus 1, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network.

  The frame extraction unit 2 extracts a predetermined number of frame images at the same time from the two moving images.

  The area dividing unit 3 performs area dividing processing on the frame image extracted by the frame extracting unit 2, and generates an area image in which area information is added to the frame image.

  The region comparison unit 4 compares the region images generated from the frame images at the same time, and generates a correspondence table in which the correspondence relationship between the two images and the color correspondence relationship for each region are recorded.

  The corresponding color learning unit 5 learns the color correspondence of the correspondence table generated by the region comparison unit 4 and obtains a color conversion rule between corresponding regions.

  The color correction moving image generation unit 6 performs color conversion on one moving image using the learning result created by the corresponding color learning unit 5.

  The color difference evaluation unit 7 evaluates the color difference between moving images after color conversion.

  The media control unit 8 reads a moving image from the media 8A.

  The display unit 9 reproduces a moving image.

  Next, the processing flow of the color correction apparatus 1 will be described.

  FIG. 2 is a flowchart showing a flow of processing in which the color correction apparatus 1 according to the present embodiment reduces the color difference between moving images. The processing of the color correction apparatus 1 can be divided into three stages: a learning phase, a correction phase, and a determination phase.

  First, the learning phase (steps ST1 to ST5) will be described.

  The media control unit 8 reads out the moving images MA and MB from the media 8A and stores them in the storage means provided in the color correction apparatus 1 (step ST1). The moving images MA and MB are obtained by two cameras of different models. The parameters that can be adjusted on the camera side such as the frame rate, the moving image size, and the time are matched, and the same subject is different at the same timing. It was taken from a viewpoint.

  Subsequently, the frame extraction unit 2 randomly extracts an appropriate number of frame images MA (t) and MB (t) at the same time from the two moving images MA and MB read (step ST2). In the example shown in FIG. 3, the frame images MA (0), MA (5), MA (9) of the moving image MA at the time t = 0, 5, 9 selected at random, and the frame images of the moving image MB. MB (0), MB (5), and MB (9) are extracted.

  Subsequently, the region dividing unit 3 performs region dividing processing on the frame images MA (t) and MB (t) extracted by the frame extracting unit 2 to generate region images RA (t) and RB (t). (Step ST3). As the region division method, various existing region division methods such as the k-means method, the Mean Shift method, and the region expansion method can be used. FIG. 4 is a diagram illustrating an example in which a region image is generated from a frame image. In the example shown in the figure, area division processing is performed on the frame images MA (0) and MB (0) shown in FIG. 4A using the Mean Shift method, and portions a to e and f to k are respectively obtained. Region images RA (0) and RB (0) shown in FIG. 4B divided into regions were generated. The area images RA (0) and RB (0) are obtained by adding information of the original frame images MA (0) and MB (0) to area number information for identifying a partial area. That is, the region images RA (0) and RB (0) are data having RGB values and region numbers for each pixel.

  Subsequently, the region comparison unit 4 compares the region images RA (t) and RB (t) generated by the region dividing unit 3, and determines the correspondence between the regions and the region correspondence between the two images based on the similarity of coordinates and colors. A correspondence table describing the correspondence between the colors is generated (step ST4). An example of the correspondence table generated by the region comparison unit 4 is shown in FIG. Details of the process for determining the correspondence will be described later.

  Subsequently, the correspondence color learning unit 5 learns the correspondence relationship of the colors in the correspondence table generated by the region comparison unit 4 and changes the color of one region from the corresponding two regions to the color of the other region. The conversion rule is obtained (step ST5). For example, when learning is performed using the error back propagation method in a feedforward type neural network, the configuration shown in FIG. 6 can be considered. When an average RGB value in a certain area of the area image RA (t) is input to the input layer 50 as input values R, G, B, it is output as output values R ′, G ′, B ′ from the output layer 52 via the intermediate layer 51. Is done. The average RGB value of the corresponding region of the region image RB (t) corresponds to the correct value (teacher value) 53. The parameter of the intermediate layer 51 is adjusted so that the error when the correct values R ″, G ″, B ″ are compared with the output values R ′, G ′, B ′ is minimized. Learning is performed by performing a combination of the correspondence relationships of the generated colors, and when there is no corresponding region like the region k in Fig. 5, the color of that region is not used for learning.

  Next, the processing content of the correction phase (step ST6) will be described.

  The color correction moving image generation unit 6 performs color conversion on the entire frame image MA (t) of the moving image MA using the learning result created by the corresponding color learning unit 5, and generates a color corrected moving image MA ′. (Step ST6).

  Next, processing contents of the determination phase (steps ST6 to ST11) will be described.

  First, the color-corresponding relationship is determined by performing the same processing as the processing in steps ST2 to ST4 on the color-corrected moving image MA 'and moving image MB (steps ST7 to ST9). Specifically, first, the frame extraction unit 2 performs random extraction of the frame images MB (t) and MA ′ (t) at the same time from the moving images MB and MA ′ (step ST7). Subsequently, the region dividing unit 3 performs region dividing processing on the extracted frame images MB (t) and MA ′ (t) to generate region images RB (t) and RA ′ (t). Further, the region comparison unit 4 determines the correspondence between the regions of the region images RB (t) and RA ′ (t) and the correspondence between the colors (step ST9).

  Then, the color difference evaluation unit 7 evaluates the color difference on the basis of the Euclidean distance of the average RGB value of each corresponding region (step ST10), and when the color difference is below a separately set threshold (No in step ST11), color correction is performed. It is assumed that the moving image MA ′ has been generated, and the process ends. When the color difference exceeds a threshold value set separately (Yes in step ST11), the process returns to step ST2, and additional learning is performed using new frame images MA (t) and MB (t). The color difference evaluation method is not limited to the above method, and other methods may be used.

  Next, the process in which the area comparison unit 4 determines the correspondence will be described. The determination of the correspondence relationship includes scanning of the area image and processing for generating the correspondence relationship from the scanning result.

  First, processing for scanning a region image will be described. FIG. 7 is a flowchart showing a flow of processing in which the region comparison unit 4 scans a region image.

  On the region images RA (t) and RB (t), an image scanning window having a horizontal width that is the maximum horizontal width size of the image and a vertical width that is a predetermined size on the basis of the coordinates (0, 0) at the upper left of the image. Arrange (step ST31). FIG. 8 shows an example of an image scanning window. In the example shown in the figure, the image size of each of the area images RA (t) and RB (t) is 320 × 240 pixels, and the window size is set to 320 × 10 pixels. The number of pixels in the window can be arbitrarily set.

  Subsequently, RGB values and area information are read out in order from the upper left pixel in the window and temporarily stored (step ST32), and the next pixel is selected so as to scan the window from the upper left to the lower right ( Step ST33). If the selected pixel is within the window (Yes in step ST34), the process returns to step ST32 to read out the RGB value and area information.

  When all the pixels in the window have been scanned (No in step ST34), the temporarily stored RGB values are averaged for each region (step ST35), and as a scanning result, region information existing in the window, The average RGB value for each area in the window and the number of areas in the window are recorded (step ST36).

  Then, the window arrangement position is moved in the y-axis direction by the number of vertical pixels of the window (step ST37). When the window is arranged in the area images RA (t) and RB (t) (Yes in step ST38), the processes in steps ST32 to ST37 are repeated. Thereby, as shown in FIG. 9, the area images RA (t) and RB (t) are scanned in order from the top.

  FIG. 10 shows a scanning result table generated by the processing shown in FIG. As shown in FIG. 10, the number of detected areas, area information, and average RGB values are recorded in the scanning result table for each of the scanning areas S1 to S24. The combination of the area information and the average RGB value is recorded as the number of areas in each of the scanning areas S1 to S24. For example, in the scanning region S3 of the region image RA (t) in FIG. 10, two regions, a region a having an average RGB value of (88, 2, 9) and a region c having an average RGB value of (90, 13, 6). Exists.

  Next, processing for generating a correspondence relationship from the scanning result table will be described. FIG. 11 is a flowchart showing a flow of processing for generating a correspondence from the scanning result table.

  The scanning area with the smallest number of areas is searched from the scanning result table (step ST39).

  When the number of regions is 1 for both the region images RA (t) and RB (t) (Yes in step ST40), it is determined that the regions have a corresponding relationship (step ST41).

  When the number of regions is not 1 in both the region images RA (t) and RB (t) (No in step ST40), as shown in FIG. 12, the average RGB between the regions of the region images RA (t) and RB (t) The Euclidean distance of the value is obtained, and it is determined that there is a correspondence in order from the closest distance below the threshold L (step ST42). When the RGB value is 8 bits (0 to 255), the threshold value L is set within the range of (0 <L ≦ 255√3). The areas where the obtained Euclidean distance exceeds the threshold L are not associated.

  In addition, when a plurality of scanning areas having the smallest total number of areas are searched and the area images RA (t) and RB (t) have the same number of areas (1: 1, 2: 2, etc.), the search is quickly performed. If the number of regions of the region images RA (t) and RB (t) is different (1: 3, 2: 3, etc.), the side with the smallest total number (this In this case, 1: 3) is preferentially processed.

  When the corresponding area is found, the average RGB value in the entire scanning result table of each corresponding area is obtained (step ST43), and the corresponding area information and the obtained average RGB value are recorded in the correspondence table (step ST44). Specifically, in FIG. 10, when it is determined that the region a and the region f have a corresponding relationship, the average RGB value of the region a is the RGB value (90, 2, 10) of the scanning region S1, and the scanning region S2. Are obtained by averaging the RGB values (90, 0, 3) and the RGB values (88, 2, 9) of the scanning region S3. Then, the region a and the region f are displayed in the corresponding column of the region of the correspondence table, and the average RGB value (90, 2, 12) of the region a and the average RGB value (80, 2, 17) of the region f are stored in the color corresponding column. Record.

  Thereafter, the determined area information is deleted from the scan result table to reduce the number of areas (step ST45). Specifically, information on the determined areas a and f is deleted from the scanning result table shown in FIG. FIG. 13 shows a scanning result table in which information on the areas a and f is deleted.

  Then, it is confirmed whether or not an undetermined area remains in the scanning result table (step ST46). If there is an undetermined area remaining, the process returns to step ST39. To do.

  In this embodiment, the average RGB value is used as the color information of the area. However, the present invention is not limited to this, and another statistical value such as a median value or a mode value may be used instead of the RGB value. Alternatively, another color space such as HSV may be used.

  Next, another color correction apparatus according to the present embodiment will be described.

  FIG. 14 is a block diagram illustrating a configuration of another color correction apparatus according to the present embodiment. The color correction device 1 shown in the figure adds a stereoscopic moving image generation unit 11 to the color correction device shown in FIG. 1 to generate a stereoscopic moving image.

  After the process of step ST11 in FIG. 2, the stereoscopic moving image generating unit 11 performs a moving image combining process on the color-corrected moving image MA ′ and the moving image MB. A stereoscopic moving image is generated in a format suitable for the viewing mode and viewing environment, such as a frame sequential method or a side-by-side method.

  As described above, according to the present embodiment, the frame images MA (t) and MB (t) at the same time are extracted from the two moving images MA and MB obtained by photographing the same subject, and the frame image MA is extracted. Region images RA (t) and RB (t) obtained by dividing (t) and MB (t) into a plurality of regions are generated, and the region correspondences between the region images RA (t) and RB (t) and for each region The color correspondence from the frame image MA (t) to MB (t) is learned based on the color correspondence, and the moving image MA is color-corrected based on the learned conversion rule. By generating the moving image MA ′, it is possible to reduce the color difference generated between the two moving images MA and MB.

  According to the present embodiment, the frame images MA ′ (t) and MB (t) at the same time are extracted from the color-corrected moving image MA ′ and the moving image MB, respectively, and the region is divided, the region correspondence relationship, the region If the color correspondence for each color is obtained, and the color difference between the corresponding colors exceeds a predetermined threshold value, additional learning is performed using new frame images MA (t) and MB (t), thereby more accurately. It is possible to reduce the color difference generated between the two moving images MA and MB.

DESCRIPTION OF SYMBOLS 1 ... Color correction apparatus 2 ... Frame extraction part 3 ... Area division part 4 ... Area comparison part 5 ... Corresponding color learning part 6 ... Color correction moving image generation part 7 ... Color difference evaluation part 8 ... Media control part 8A ... Media 9 ... Display Unit 11... 3D image generation unit 50... Input layer 51... Intermediate layer 52.

Claims (9)

Input means for inputting two moving images of the same subject and storing them in the storage means;
Extraction means for extracting a frame image at the same time from each of the two moving images stored in the storage means;
Area dividing means for classifying each pixel of each of the frame images into one of a plurality of areas;
A region determination unit that determines a corresponding region between the frame images and obtains a color correspondence for each region;
Learning means for learning a conversion rule to be the other color when one color is converted based on the color correspondence;
Color correction means for correcting one color of the moving image using the conversion rule;
A color correction apparatus comprising: The area determination means includes
Scanning each frame image by setting a scanning area for each frame image;
For each scanning region, obtain the region existing in the scanning region and the color of the region,
The color correction apparatus according to claim 1, wherein the corresponding region is determined by comparing the color of the region existing in the scanning region between the corresponding scanning regions of each of the frame images. The area determination means includes
For each scanning area, create a scanning result table that describes the area present in the scanning area and the color of the area,
The scanning area of the scanning result table is searched, and the Euclidean color of the area between the corresponding scanning areas in order from the scanning area having the smallest number of the areas existing in the corresponding scanning area of each of the frame images. The color correction apparatus according to claim 2, wherein a distance is obtained, and an area where the Euclidean distance is the shortest with a predetermined threshold or less is determined as a corresponding area.   A frame image at the same time is extracted from each of the moving image corrected by the color correcting means and the other moving image to obtain a color correspondence relationship. If the color difference between corresponding colors exceeds a predetermined threshold, additional learning is performed. The color correction apparatus according to claim 1, further comprising a determination unit that performs the determination. Inputting two moving images of the same subject and storing them in the storage means;
Extracting a frame image at the same time from each of the two moving images stored in the storage means;
Classifying each pixel of each of the frame images into one of a plurality of regions;
Determining a corresponding region between the frame images;
Obtaining a color correspondence for each corresponding region;
Learning a conversion rule to be the other color when one color is converted based on the color correspondence;
Correcting one color of the moving image using the conversion rule;
A color correction method comprising: Determining the corresponding region comprises:
Scanning each frame image by setting a scanning area for each frame image;
For each scanning region, obtain the region existing in the scanning region and the color of the region,
The color correction method according to claim 5, wherein the corresponding region is determined by comparing the color of the region existing in the scanning region between the corresponding scanning regions of each of the frame images. Determining the corresponding region comprises:
For each scanning area, create a scanning result table that describes the area present in the scanning area and the color of the area,
The scanning area of the scanning result table is searched, and the Euclidean color of the area between the corresponding scanning areas in order from the scanning area having the smallest number of the areas existing in the corresponding scanning area of each of the frame images. The color correction method according to claim 6, wherein a distance is obtained, and an area where the Euclidean distance is the shortest with a predetermined threshold or less is determined as a corresponding area.   A frame image at the same time is extracted from each of the corrected moving image and the other moving image to obtain a color correspondence, and when the color difference between corresponding colors exceeds a predetermined threshold, a step of performing additional learning is performed. The color correction method according to claim 5, further comprising:   A color correction program that causes a computer to execute the color correction method according to claim 5.

Images