JP5900278B2 - Image processing apparatus and method - Google Patents

Description

  The present invention corrects a screen in which the brightness of an image becomes discontinuous by a flash of another person other than the photographer at the time of shooting a moving image, and can add a pseudo flash emission effect. The present invention relates to a processing apparatus and method.

  2. Description of the Related Art In recent years, many complementary metal oxide semiconductor (CMOS) sensors have been employed in imaging devices such as video cameras. Reading of the imaging signal in the CMOS sensor is performed by a rolling shutter system in which the readout timing is sequentially shifted in the line direction by sequentially performing reset, exposure, and readout in units of lines, not at the same timing in all pixels.

  For this reason, for example, when a large number of video cameras or still cameras photograph the same subject at the same time, such as at a wedding ceremony, CMOS is emitted during the exposure period of only a part of the lines, where the flashing period of the other person's flash is short and intense light is emitted. May enter the sensor. In this case, the exposure amount differs in a part of the screen to be imaged, and a white band-like bright area is generated in the screen, resulting in a discontinuous image. This problem is not limited to the flash, but also occurs when strong light having a short light emission period similar to the flash is incident on the CMOS sensor.

  Various solutions to this problem have been proposed as described in Patent Documents 1 to 3, for example.

JP 2010-213220 A JP 2011-15222 A JP 2011-101208 A

  When the portion where the brightness of the image is discontinuous due to the flash emission is corrected, the image is in a state where the flash is not substantially emitted. However, there is a desire to intentionally maintain an image in a state where the flash is emitted so as not to impair the sense of reality that the image was taken in a state where the flash is emitted.

  At this time, it is required to detect and correct as little as possible misdetection that the brightness of the image is discontinuous in the screen due to the light having a short light emission period such as a flash entering the CMOS sensor. Further, it is not preferable that the memory resources increase in order to detect that the brightness of the image is discontinuous in the screen. It is required to detect that the brightness of an image becomes discontinuous without increasing memory resources.

  In order to meet such a demand, the present invention can correct a portion where the brightness of an image is discontinuous due to flash emission, and intentionally keep an image in a state where the flash is emitted. An object of the present invention is to provide an image processing apparatus and method that can be used. In addition, the image processing apparatus and method that can accurately detect that the brightness of the image is discontinuous in the screen, and the brightness of the image in the screen is not increased without increasing memory resources. It is an object of the present invention to provide an image processing apparatus and method capable of detecting the fact that it has become continuous.

  In order to solve the above-described problems of the conventional technology, the present invention provides a flash detection unit (2) that detects whether or not a screen that is composed of image data includes a line that is bright due to flash emission. , 20), a holding unit (4) that holds a past screen of the image data, and a line having high luminance is detected by the flash detection unit, at least high luminance is detected. A correction processing unit (82) that corrects image data by replacing the existing line with the line of the past screen held in the holding unit, and at least one of the screens corrected by the correction processing unit And an effect processing unit (83) for adding a pseudo flash emission effect to the image data constituting the effect addition target screen, and the effect addition target screen. To provide an image processing apparatus characterized by comprising a recording unit for recording image data effect is added pseudo-flash and (11) by the processing unit.

  In addition, in order to solve the above-described problems of the related art, the present invention detects whether or not a line having high luminance due to flash emission is included in a screen composed of image data, and the image If the past screen of data is held in the holding unit (4) and it is detected that a line with high luminance is included, at least the line with high luminance is held in the holding unit. The image data is corrected by replacing with the lines of the past screen, and at least one of the corrected screens is set as an effect addition target screen, and a pseudo flash is applied to the image data constituting the effect addition target screen. There is provided an image processing method characterized in that image data to which a light emission effect is added and a pseudo flash light emission effect is added is recorded in a recording unit (11).

  According to the image processing apparatus and method of the present invention, it is possible to correct a portion where the brightness of an image becomes discontinuous due to light emission of the flash, and to intentionally keep an image in a state where the flash is emitted. it can. Further, according to the image processing apparatus and method of the present invention, it is possible to accurately detect that the brightness of the image is discontinuous within the screen, and without increasing the memory resources, Can detect that the brightness of the image is discontinuous.

1 is a block diagram illustrating an image processing apparatus according to a first embodiment. It is a figure for demonstrating the phenomenon in which the brightness of the image which generate | occur | produces by light emission of a flash at the time of imaging | photography of a moving image using a CMOS sensor becomes discontinuous. It is a flowchart which shows the whole flow of imaging | photography and the recording process in the image processing apparatus of 1st Embodiment. It is a flowchart which shows the detail of the flash detection process in 1st Embodiment. It is a flowchart which shows the detail of the flash correction process by 1st Embodiment. It is a figure for demonstrating correction | amendment of the image data in the case of the screen unit switching mode in 1st Embodiment. It is a figure which shows the adjacent area | region for demonstrating the operation | movement by 1st Embodiment. It is a figure which shows the mixing characteristic for demonstrating the operation | movement by 1st Embodiment. It is a figure for demonstrating correction | amendment of the image data in the case of the area | region correction mode in 1st Embodiment. It is a block diagram which shows the image processing apparatus of 2nd Embodiment. It is a flowchart which shows the detail of the flash detection process in 2nd Embodiment. It is a figure for demonstrating the effect by 2nd Embodiment. It is a flowchart which shows the whole flow of imaging | photography and the recording process in the image processing apparatus of 3rd Embodiment. It is a flowchart which shows the detail of the flash detection and correction process in 3rd Embodiment.

  Embodiments of an image processing apparatus and method according to the present invention will be described below with reference to the accompanying drawings. In each embodiment, an image processing method applied to a video camera and applied to the video camera will be described as an example.

<First Embodiment>
The overall configuration of a video camera 101 that is an image processing apparatus according to the first embodiment will be described with reference to FIG. The video camera 101 includes an imaging unit 1, a flash detection unit 2, an image signal processing unit 3, a buffer memory 4, a compression / decompression processing unit 5, an output signal processing unit 6, a display unit 7, a flash correction unit 80, a control unit 9, a memory 10 includes a recording / reproducing unit 11 and an operation switch 12. The recording / reproducing unit 11 has an arbitrary recording medium such as a semiconductor memory or a hard disk drive. The recording medium may be detachable from the video camera 101.

  The imaging unit 1 includes an optical system (not shown) such as a lens and an aperture, a CMOS sensor 1s, an exposure and focus control mechanism (not shown), and the like. As an example, the imaging unit 1 includes three CMOS sensors 1s, and outputs electrical signals indicating R, G, and B signals that are red, green, and blue color signals from the three CMOS sensors 1s. Of course, the number of CMOS sensors 1s may be one. The imaging unit 1 extracts optical information input to the CMOS sensor 1 s through the lens as an electrical signal at a predetermined timing, and supplies the electrical signal to the flash detection unit 2 and the image signal processing unit 3.

  The flash detection unit 2 includes a line average luminance calculation unit 21, a line average luminance holding unit 22, a full screen average luminance calculation unit 23, a full screen average luminance holding unit 24, a difference calculation unit 25, and a flash determination unit 26. The flash detection unit 2 detects the flash emission according to the control by the control unit 9 as follows.

  The line average luminance calculation unit 21 calculates the average brightness for each line in the electrical signal supplied from the imaging unit 1. In the first embodiment, the average brightness for each line is calculated based on the G signal among the R, G, and B signals. Specifically, the line average luminance calculating unit 21 adds the value of the G signal over the effective pixels of one line, and divides by the number of effective pixels to average the brightness of one line (line average luminance). Is calculated. The signal based on which the line average luminance is calculated is not limited to the G signal.

  The line average luminance calculated by the line average luminance calculation unit 21 is input to the line average luminance holding unit 22 and the full screen average luminance calculation unit 23. The line average luminance holding unit 22 temporarily holds the line average luminance calculated by the line average luminance calculating unit 21. The entire screen average luminance calculation unit 23 calculates the average brightness of one screen (one frame). Specifically, the entire screen average luminance calculation unit 23 adds the average line luminance sequentially input over the effective lines of one screen, and divides by the number of effective lines, thereby averaging the brightness of the entire one screen ( The average luminance of the entire screen) is calculated. The full screen average luminance holding unit 24 temporarily holds the full screen average luminance.

  The difference calculating unit 25 receives the line average luminance held by the line average luminance holding unit 22 and the full screen average luminance held by the full screen average luminance holding unit 24. Note that the screen for which the average average luminance is calculated is different from the screen for which the average average luminance is calculated by one screen. The screen for which the average average luminance is calculated is the screen one screen before the screen for which the average average luminance is calculated. is there. The entire screen average brightness of the previous screen may be retained for one screen period and used as the entire screen average brightness of the previous two screens. Also, if both the screen average brightness of the previous screen and the screen average brightness of the previous two screens are retained and the previous screen is affected by the flash emission, You may make it use the whole screen average brightness | luminance of a previous screen.

  In the first embodiment, the screen on which the line average brightness is calculated is the current screen, and the screen on which the total screen average brightness is calculated is the previous screen. That is, the current screen is a target screen that is a target for detection as to whether or not it includes a line that is bright due to flash emission. The difference calculation unit 25 sequentially calculates the difference between each line average luminance on the current screen and the entire screen average luminance on the previous screen. The difference calculation unit 25 takes the absolute value of the difference and outputs it as a difference value.

  The difference value calculated by the difference calculation unit 25 is input to the flash determination unit 26. The flash determination unit 26 determines whether or not the difference value for each line is equal to or greater than a set threshold value, and determines whether or not the state where the difference value is equal to or greater than the threshold value continues for a preset number of lines. . When it is determined that the state where the difference value is equal to or greater than the threshold value continues for a preset number of lines or more, the flash determination unit 26 detects that the brightness of the image is discontinuous due to flash emission.

  The flash determination unit 26, together with the detection signal indicating that the flash is emitted, the data indicating the first line number when the difference value at the start of flash emission is equal to or greater than the threshold, and the difference value is less than the threshold. The data indicating the first line number is output. The line one line before the first line where the difference value is less than the threshold is the last line of flash emission.

  For example, the flash determination unit 26 sets “1” as a detection signal indicating that the flash is emitted when it is determined that the flash is emitted, and the flash is emitted when it is determined that the flash is not emitted. What is necessary is just to produce | generate "0" as a detection signal which shows not having existed.

  As can be seen from the above description, the flash detection unit 2 only calculates line average brightness, full screen average brightness, and difference value, and determines the difference value to detect flash emission. It is not necessary to provide a large memory resource. Note that although the flash emission is detected here, as described above, the flash detection unit 2 also has a discontinuous image brightness even when strong light having a short emission period is incident on the CMOS sensor 1s. It detects that it became. Including such a case, the flash emission is detected.

  The flash detection unit 2 compares the line average brightness of each line on the current screen with, for example, the screen average brightness of the past screen which is the previous screen, so that some lines of the screen are flashed. When the luminance is high due to light emission, the difference value is equal to or greater than the threshold value for all lines having high luminance. If the average line brightness of adjacent lines in the current screen is compared, the difference value is equal to or greater than the threshold value only at the boundary between the normal line and the line with high brightness. Therefore, in the present embodiment, it is possible to detect that the brightness of the image is discontinuous on the screen with few false detections.

  The image signal processing unit 3 converts the electrical signal supplied from the imaging unit 1 into a predetermined signal system. As described above, in the first embodiment, since the electrical signals are R, G, and B signals, after adjusting the white balance and gain and performing gamma processing, the luminance signal Y and the color difference signals Cb, Convert to Cr. The flash detection unit 2 may calculate the line average luminance and the full screen average luminance based on the luminance signal Y. The buffer memory (holding unit) 4 temporarily holds the image data of the luminance signal Y and the color difference signals Cb and Cr output from the image signal processing unit 3. If the color difference signals Cb and Cr are band-limited and the luminance signal Y is time-axis multiplexed, memory and calculation resources necessary for image processing can be reduced.

  When the flash emission is detected, the image data input from the image signal processing unit 3 to the buffer memory 4 and held therein is used in the image data correction processing by the flash correction unit 80 described later.

  The compression / decompression processing unit 5 compresses the image data read from the buffer memory 4 by a compression method such as MPEG-2, MPEG-4 AVC / H.264, JPEG, etc. to generate encoded data, and also records and reproduces the data. When the encoded data recorded in 11 is reproduced and read, the encoded data is decompressed.

  The output signal processing unit 6 outputs the image data read from the buffer memory 4 to an external device (not shown) and converts it to, for example, the NTSC signal system in accordance with the input format of the display unit 7 at the subsequent stage. The size of the image data is changed according to the size of the display screen. As a result, the image data becomes data that can be displayed on the display unit 7 and is supplied to the display unit 7 for display. The on-screen signal output from the control unit 9 is also input to the output signal processing unit 6, and the output signal processing unit 6 superimposes the on-screen signal on the image data.

  The detection signal output from the flash determination unit 26 is input to the flash correction unit 80. The flash correction unit 80 includes a correction control unit 81, a correction processing unit 82, and an effect processing unit 83.

  The flash correction unit 80 corrects image data whose brightness has become discontinuous due to flash emission under the control of the control unit 9. The flash correction unit 80 corrects the image data when reproducing the image data when image data that is partially bright due to flash emission is recorded in the recording / reproduction unit 11. The operation of the correction processing unit 82 when the region correction mode is set and when the screen unit switching mode is set will be described later.

  The effect processing unit 83 adds a pseudo flash effect to the image data corrected by the correction processing unit 82 and supplies the image data to the buffer memory 4.

  The control unit 9 is set with either an area correction mode or a screen unit switching mode to be described later. The flash detection unit 2 and the flash correction unit 80 refer to the mode information set in the control unit 9. The correction control unit 81 controls the correction processing unit 82 so that the correction processing unit 82 executes a correction processing operation corresponding to the mode set in the control unit 9. Note that the screen unit switching mode is not essential, and the region correction mode and the screen unit switching mode may be selected as necessary.

  The control unit 9 can be configured by a microcomputer having a CPU. The control unit 9 has a storage unit that stores a control program for controlling the entire video camera 101, and controls the entire video camera 101 based on the control program. When the image processing according to the first embodiment is realized by an image processing program, an image processing program for executing the image processing according to the first embodiment may be included as a part of the control program. In this case, the control unit 9 executes the image processing program to detect flash emission, and corrects the image data when the flash emits light.

  A memory 10 as a working memory of the control unit 9 is connected to the control unit 9. In addition, a recording / reproducing unit 11 and an operation switch 12 are connected to the control unit 9. When the control unit 9 is instructed to record the moving image captured by the operation switch 12, the control unit 9 controls to record the encoded data output from the compression / decompression processing unit 5 in the recording / reproducing unit 11.

  Further, when an instruction to record a moving image captured by the operation switch 12 is given, the control unit 9 controls the recording / reproducing unit 11 to reproduce and read out the encoded data and supply it to the compression / decompression processing unit 5. The control unit 9 generates an on-screen signal according to an operation by the operation switch 12 and supplies it to the output signal processing unit 6.

  Although FIG. 1 shows an example in which the flash detection unit 2 is configured by hardware, it may be configured by software. That is, a configuration equivalent to the flash detection unit 2 may be realized by executing a flash detection program that is a part of the image processing program of the first embodiment. The flash correction unit 80 may also be configured by software. That is, a configuration equivalent to the flash correction unit 80 may be realized by executing a flash correction program that is a part of the image processing program of the first embodiment.

  Here, with reference to FIG. 2, a phenomenon in which the brightness of the image that occurs when the flash of another person other than the photographer emits light when shooting a moving image is discontinuous will be described. Since the CMOS sensor 1s adopting the rolling shutter system reads out pixel data imaged in units of line Ln, subject imaging ST1, ST2, ST3, ST4... By the imaging unit 1 is as shown in FIG. The timing sequentially shifts in the time direction from the first line Ln to the last line Ln on the screen. The imaging of one line Ln includes an exposure time and a pixel data readout time, as shown in FIG.

  As shown in (c) of FIG. 2, the effective data D1, D2, D3, D4... Are output from the imaging unit 1 by the imaging ST1, ST2, ST3, ST4. It is assumed that there is a light emission FL caused by another person's flash at time t1 and enters the CMOS sensor 1s. In this case, each line Ln from the position indicated by the alternate long and short dash line to the last line Ln and each line Ln from the first line Ln to the position indicated by the alternate long and short dash line of the imaging ST3 are affected by the light emission FL. Become.

  If the image data correction according to the first embodiment is not performed, the images Im1, Im2, Im3, Im4,... Recorded and reproduced in the recording / reproducing unit 11 and displayed on the display unit 7 are shown in FIG. As shown. The images Im1 and Im4 are not affected by the light emission FL, and the screen brightness is normal continuously. The image Im2 is an abnormal image in which the lower end portion after the one-dot chain line is brightened in a white band shape, and the brightness of the image is discontinuous. A hatched portion indicates a portion with high luminance. The image Im3 is an abnormal image in which the upper end portion from the upper end to the alternate long and short dash line is similarly brightened in a white band shape, and the brightness of the image becomes discontinuous.

  Note that when the flash of the video camera 101 owned by the photographer is emitted, the flash emission timing is matched with the imaging timing shown in FIG. 2B, and as shown in FIG. The phenomenon that the brightness of the image becomes discontinuous does not occur.

  The operation of the video camera 101 according to the first embodiment will be described in more detail with reference to FIGS. FIG. 3 shows the overall flow of shooting and recording processing by the video camera 101. In FIG. 3, the control unit 9 determines whether or not there is an instruction to start shooting with the operation switch 12 in step S <b> 1. If it is determined that there is an instruction to start shooting (YES), the control unit 9 starts shooting by controlling the imaging unit 1 in accordance with conditions such as an aperture and a shutter speed in step S2. Note that an instruction to turn on the video camera 101 by the operation switch 12 may be an instruction to start shooting. If it is not determined that there is an instruction to start shooting (NO), the control unit 9 returns the process to step S1 and repeats step S1.

  In step S30, the control unit 9 causes the flash detection unit 2 to execute flash detection processing. Further, in parallel with step S30, the control unit 9 causes the image signal processing unit 3 to execute image signal processing in step S4. Details of the flash detection processing in step S30 will be described later. In step S5, the control unit 9 controls the output signal processing unit 6 to display the captured image on the display unit 7.

  In step S <b> 6, the control unit 9 determines whether or not there is an instruction to start recording by the operation switch 12. If it is determined that there is an instruction to start recording (YES), the control unit 9 causes the recording / reproducing unit 11 to record the encoded data output from the compression / decompression processing unit 5 in step S7. If it is not determined that there is an instruction to start recording (NO), the control unit 9 shifts the process to step S11.

  In step S8, the controller 9 determines whether or not there is an instruction to end recording. If it is determined that there is an instruction to end recording (YES), the control unit 9 stops recording in the recording / reproducing unit 11 in step S9. If it is not determined that there is an instruction to end recording (NO), the control unit 9 returns the process to step S7 and repeats step S7.

  Following step S9, the control unit 9 determines in step S11 whether or not there has been an instruction to start reproduction of predetermined image data recorded in the recording / reproducing unit 11 by the operation switch 12. If it is determined that there is an instruction to start reproduction (YES), the control unit 9 reproduces the image data (encoded data) recorded in the recording / reproducing unit 11 in step S12. The encoded data read from the recording / reproducing unit 11 is decompressed by the compression / decompression processing unit 5 and input to the buffer memory 4 to be held. If it is not determined that there is an instruction to start reproduction (NO), the control unit 9 shifts the process to step S10.

  In step S13, the controller 9 determines whether there is a flash detection area. The flash detection area is an area in which high brightness is detected by flash emission in the image data to be reproduced. As will be described later, which part of the image data is the flash detection area can be determined by referring to metadata (additional information) added to the encoded data.

  If it is determined that there is a flash detection area (YES), the control unit 9 causes the flash correction unit 80 to execute flash correction processing in step S140. The flash correction process in step S140 is a correction process for correcting a portion where the brightness of the image is discontinuous due to flash emission and adding a pseudo flash emission effect.

  In step S15, the control unit 9 causes the display unit 7 to display the image data corrected in step S140.

  In this case, the decompressed image data held in the buffer memory 4 is input to the flash correction unit 80, and the flash correction process in step S140 is executed. The corrected image data is input and held in the buffer memory 4 and supplied to the display unit 7 through the output processing unit 6.

  If it is not determined in step S13 that there is a flash detection area (NO), the control unit 9 shifts the process to step S15. In step S15, the control unit 9 causes the display unit 7 to display the image data reproduced in step S12 as it is. In this case, the decompressed image data held in the buffer memory 4 is directly output from the buffer memory 4 to the output processing unit 6 and supplied to the display unit 7.

  In step S16, the controller 9 determines whether or not an instruction to stop reproduction has been issued. If it is determined that there is an instruction to stop reproduction (YES), the control unit 9 stops reproduction of the image data in step S17, and shifts the processing to step S10. If it is not determined that there has been an instruction to stop playback (NO), the control unit 9 returns the process to step S12 and repeats step S12 and subsequent steps.

  In step S <b> 10, the control unit 9 determines whether or not an instruction to end shooting is given by the operation switch 12. If it is determined that there is an instruction to end shooting (YES), the control unit 9 ends the process. If it is not determined that there is an instruction to end shooting (NO), the control unit 9 returns the process to step S2 and repeats step S2 and subsequent steps. An instruction to turn off the power of the video camera 101 by the operation switch 12 may be used as an instruction to end shooting.

Details of the flash detection process in step S30 shown in FIG. 3 will be described with reference to FIG. The step S30 in the first embodiment and step S30 _1. The following steps are executed by the flash detection unit 2 based on control by the control unit 9.

  In FIG. 4, in step S3101, the flash detection unit 2 sequentially executes a process of adding pixel values of effective pixels in the effective line for each effective line. In step S3102, the flash detection unit 2 calculates the line average luminance by dividing the addition value of the pixel values obtained in step S3101 by the number of effective pixels, and holds the line average luminance.

  In step S3103, the flash detection unit 2 executes a process of sequentially adding the line average luminance. In step S3104, the flash detection unit 2 executes a difference calculation process between the line average luminance calculated in step S3102 and the average luminance of the entire screen one screen before. The difference calculation process in step S3104 is executed every time the line average luminance is calculated in step S3102.

  In step S3105, the flash detection unit 2 determines whether or not the flash is in a normal state in which no flash emission is detected. If it is determined that the current state is the normal state (YES), in step S3106, the flash detection unit 2 determines whether a line having a difference value equal to or larger than the threshold value is detected. If it is determined that a line having a difference value equal to or greater than the threshold value is detected (YES), the flash detection unit 2 moves the process to step S3107. At this time, the flash detection unit 2 stores detection line information indicating how many lines are detected from the first line of the frame.

  On the other hand, if it is not determined that a line having a difference value equal to or greater than the threshold value has been detected (NO), the flash detection unit 2 deletes the detected line information if it has already been stored, and the process proceeds to step S3111. Transition.

  In step S3107, the flash detection unit 2 determines whether or not the state where the difference value is equal to or greater than the threshold value continues for a predetermined number of lines or more. If it is determined that the predetermined number of lines or more have been continued (YES), the flash detection unit 2 shifts the process to step S3108. If it is not determined that the predetermined number of lines or more has been continued (NO), the flash detection unit 2 The process proceeds to step S3111.

  In step S3105, the normal state is determined. In step S3106, a line having a difference value equal to or greater than the threshold value is detected. In step S3107, it is determined that the state in which the difference value is equal to or greater than the threshold value continues for a predetermined number of lines. In this case, it is detected that the state has changed from a state not affected by the light emission of the flash to a state of high brightness due to the light emission of the flash.

  Therefore, in step S3108, the flash detection unit 2 changes the flash detection state from the state where the flash emission is not detected to the state where the flash emission is detected. In the first embodiment, the flash detection unit 2 supplies detection line information indicating the flash emission start line to the control unit 9 and erases the stored detection line information.

  On the other hand, if it is not determined in step S3105 that the current state is the normal state (NO), the flash detection unit 2 determines in step S3109 whether a line having a difference value less than the threshold value has been detected. If it is determined that a line having a difference value less than the threshold value is detected (YES), the flash detection unit 2 moves the process to step S3110. At this time, the flash detection unit 2 stores detection line information indicating how many lines are detected from the first line of the frame.

  In step S3110, the flash detection unit 2 determines whether or not the state where the difference value is less than the threshold value continues for a predetermined number of lines or more. If it is determined that the predetermined number of lines or more have been continued (YES), the flash detection unit 2 shifts the process to step S3108. If it is not determined that the predetermined number of lines or more has been continued (NO), the flash detection unit 2 The process proceeds to step S3111.

  In step S3105, it is determined that the current state is not the normal state. In step S3109, a line having a difference value less than the threshold value is detected. In step S3110, it is determined that the state in which the difference value is less than the threshold value continues for a predetermined number of lines. In this case, it is detected that the state has changed from a high brightness state to a state not affected by the light emission of the flash under the influence of the light emission of the flash.

  Therefore, in step S3108, the flash detection unit 2 changes the flash detection state from the state in which the flash emission is detected to the state in which the flash emission is not detected. In the first embodiment, the flash detection unit 2 supplies detection line information indicating the end line of flash emission to the control unit 9 and erases the stored detection line information.

  In step S3111, the flash detection unit 2 determines whether the determination for all active lines is completed. If it is not determined that the determination for all active lines has been completed (NO), the flash detection unit 2 returns the process to step S3101 and repeats the processes after step S3101. If it is determined that the determination for all the effective lines has been completed (YES), the flash detection unit 2 calculates the total screen average luminance by dividing the added value of the line average luminance by the number of effective lines in step S3112. Holds the full screen average brightness.

  When the control unit 9 records the encoded data output from the compression / decompression processing unit 5 in the recording / reproducing unit 11 in step S7 of FIG. 3, the start line and end of the flash emission detected by the flash detection unit 2 are recorded. Detection line information indicating a line is recorded as metadata to be added to the encoded data.

  Details of the flash correction processing in step S140 shown in FIG. 3 will be described with reference to FIG. The following steps are executed by the flash correction unit 80 based on control by the control unit 9. The control unit 9 causes the flash correction unit 80 to execute flash correction processing based on the metadata added to the encoded data.

  In FIG. 5, the flash correction unit 80 determines whether or not the screen unit switching mode is set in step S1401. If it is determined that the screen unit switching mode is set (YES), the flash correction unit 80 determines whether or not the flash is for two screens in step S1402.

  If it is determined that the flash covers two screens (YES), the flash correction unit 80 determines whether or not the screen is the second screen in step S1403. If it is determined that the screen is the second screen (YES), the flash correction unit 80 replaces the screen after the first screen in step S1404. If it is not determined in step S1402 that the flash extends over two screens (NO), or if it is not determined in step S1403 that the second screen is displayed (NO), the flash correction unit 80 determines in step S1405. Replace with the previous screen.

  6A shows images Im1, Im2, Im3, Im4,... Before correction. In FIG. 6, the hatched portion is a portion having high brightness due to flash emission. As shown in FIG. 6B, the image Im2 of the first screen having the high luminance portion is replaced with the image Im1 of the previous screen in step S1405, and the second screen having the high luminance portion. In step S1404, the image Im3 is replaced with the image Im4 after one screen.

  Next, in step S1410, the flash correction unit 80 (effect processing unit 83) adds a pseudo flash emission effect by adding a luminance signal of a predetermined value or more to the entire corrected screen. The luminance signal may be a fixed value.

  Instead of adding a fixed value of the luminance signal to the image data, a pseudo flash emission effect is added by multiplying the luminance signal or both the luminance signal and the color difference signal by a predetermined gain. May be. When multiplying both the luminance signal and the color difference signal by a gain, it is preferable to individually set a gain for multiplying the luminance signal and a gain for multiplying the color difference signal.

  FIG. 6C shows the effect of the pseudo flash emission on the entire screen in which the image Im2 is replaced with the image Im1 of the previous screen, and on the entire screen in which the image Im3 is replaced with the image Im4 of the next screen. The added state is shown. The images Im1 and Im4 are images Im1 ″ and Im4 ″ that are equivalent to the state in which the flash is emitted on the entire screen, as shown by hatching the entire screen.

  In the above description, the pseudo flash emission effect is added to the two images (Im2, Im3) determined to have a high-luminance portion by flash emission after replacing them with Im1 and Im4, respectively. It is also possible to add a pseudo flash emission effect using only one of the images Im2 and Im3 (screen) as an effect addition target screen and perform only replacement on the remaining image (screen).

  On the other hand, if it is not determined in step S1401 that the screen unit switching mode is set (NO), the flash correction unit 80 executes processing in the case where the region correction mode is set in steps S1406 to S1408. In step S1406, the flash correction unit 80 divides one screen into a plurality of areas.

  Specifically, as shown in FIG. 7A, the flash correction unit 80 is positioned above the flash start line Lnfs indicated by the detection line information detected in the flash detection process of step S30, and the flash start line. A plurality of adjacent areas Aadjs adjacent to Lnfs and an adjacent area Aadje positioned below the flash end line Lnfe indicated by the detection line information and adjacent to the flash end line Lnfe as shown in FIG. Set. The adjacent areas Aadjs and Aadje may be about 5 to 10 lines, for example.

  As described above, the flash correction unit 80 is adjacent to a normal part that is not affected by the flash, a part that has high brightness due to flash emission, and an upper side or a lower side of the part that has high brightness. The area is divided into adjacent areas.

  In step S1407, the flash correction unit 80 generates a composition ratio. Specifically, in FIG. 7A, when the first line of the adjacent area Aadjs is LS, the last line of the adjacent area Aadjs is LE, and the pixel position of the adjacent area Aadjs is Lf (i, j), The composition weighting coefficient Wf (i, j) of the image data of the current screen and the composition weighting coefficient Wb (i, j) of the image data of the screen one screen before are expressed by Expression (1).

  In Expression (1), the pixel position Lf (i, j) is used, but the image data of the previous screen and the image data of the current screen are mixed in units of lines, and thus indicate the horizontal position of the image. There is no need for i information.

  In FIG. 7B, assuming that the first line of the adjacent area Aadje is LS, the last line of the adjacent area Aadje is LE, and the pixel position of the adjacent area Aadje is Lf (i, j), the current screen image The composition weighting coefficient Wf (i, j) of data and the composition weighting coefficient Wb (i, j) of the image data of the screen one screen before are expressed by Expression (2).

  Similarly, in the expression (2), the pixel position Lf (i, j) is used. However, since the image data of the previous screen and the image data of the current screen are mixed in line units, There is no need for i information indicating the position.

  FIG. 8A shows the mixing characteristics of the current screen line and the previous screen line according to the equation (1). In FIG. 8A, the solid line indicates the mixing ratio of the lines on the current screen, and the broken line indicates the mixing ratio of the lines on the previous screen. As shown in FIG. 8A, in the case of the image Im2 shown in FIG. 7A, the line from the upper end of the screen to the front of the adjacent area Aadjs is only the current screen.

  In the adjacent area Aadjs, a new line is generated by mixing the line on the current screen and the line on the previous screen based on Expression (1). From the first line LS to the last line LE of the adjacent area Aadjs, the line mixing ratio of the current screen is sequentially decreased, while the line mixing ratio of the screen of the previous screen is sequentially increased. After the last line LE, only the screen line one screen before is displayed. Note that the flash start line Lnfs and the last line LE may be the same line.

  FIG. 8B shows the mixing characteristics of the current screen line and the previous screen line according to the equation (2). In FIG. 8B, the solid line indicates the mixing ratio of the lines on the current screen, and the broken line indicates the mixing ratio of the lines on the previous screen. As shown in FIG. 8 (b), in the case of the image Im3 shown in FIG. 7 (b), the line from the upper end of the screen to the front of the adjacent area Aadje is only the screen line one screen before.

  In the adjacent area Aadje, based on the formula (2), a line on the previous screen and a line on the current screen are mixed to generate a new line. From the first line LS to the last line LE of the adjacent area Aadje, the line mixing ratio of the screen one previous screen is sequentially decreased, while the line mixing ratio of the current screen is sequentially increased. After the last line LE, there are only lines on the current screen. The flush end line Lnfe and the first line LS may be the same line.

  According to the first embodiment, when the brightness is high from the middle of the screen as in the image Im2 shown in FIG. 2, the adjacent to the flash start line Lnfs is not affected by the flash located above. In the area Aadjs, the screen line is gradually switched to the screen on the previous screen as it progresses from the first line LS to the last line LE. Therefore, the unnaturalness of mixing the image data of the current screen and the image data of the previous screen is reduced.

  Further, according to the first embodiment, when the image is a high-brightness and normal image from the middle like the image Im3 shown in FIG. 2, it is positioned below the flash end line Lnfe. In the adjacent area Aadje that is not affected by the flash, the line is gradually switched to the line on the current screen as it proceeds from the first line LS to the last line LE. Therefore, the unnaturalness of mixing the image data of the current screen and the image data of the previous screen is reduced.

  Instead of the above formulas (1) and (2), the image data of the current screen and the image data of the previous screen may be mixed in units of pixels using the formula (3). In Expression (3), Mf (i, j) is the brightness of the pixel, and ThH and ThL are brightness thresholds that satisfy ThH> ThL. In this case, the brightness of each effective pixel detected by the flash detection unit 2 may be input to the flash correction unit 80.

  According to Expression (3), if the brightness Mf (i, j) of each pixel is smaller than the threshold value ThL of the smaller value in the adjacent areas Aadjs and Aadje, the composite weighting coefficient of the image data of the current screen Wf (i, j) becomes 1, and the composite weighting coefficient Wb (i, j) of the image data of the previous screen becomes 0. That is, if the pixel brightness Mf (i, j) is smaller than the threshold ThL, there is a high possibility that the pixel is not affected by the flash emission, and only the pixel data of the current screen is used.

  If the brightness Mf (i, j) of each pixel is larger than the larger threshold value ThH, the composite weighting coefficient Wf (i, j) of the image data of the current screen becomes 0, and the screen one screen before The composite weighting coefficient Wb (i, j) of the image data is 1. That is, if the pixel brightness Mf (i, j) is larger than the threshold value ThH, there is a high possibility that it is affected by the light emission of the flash, and only the pixel data of the previous screen is used.

  If the brightness Mf (i, j) of each pixel is greater than or equal to the threshold ThL and less than or equal to the threshold ThH, the pixel data of the current screen and the pixel data of the previous screen are adaptively represented by the characteristic of Equation (3). Will be mixed.

  In this way, in step S1407, using Formulas (1) and (2), the composition ratio (mixing rate) of the current screen line and the previous screen line is generated, or Formula (3) is By using this, a composition ratio (mixing ratio) between the image data of the current screen and the image data of the previous screen is generated in units of pixels.

  In step S1408, the flash correction unit 80 combines the current screen line (pixel) and the previous screen line (pixel) by the combination ratio generated in step S1407, and executes region replacement processing.

  When the area correction mode is set in the control unit 9, as shown in (a) of FIG. 9, a plurality of lines having high luminance in the hatched portion in the image Im2 are displayed in the image Im1. Replaced with corresponding lines. In addition, a plurality of lines with high luminance in the hatched portions in the image Im3 are replaced with corresponding lines in the image Im2. In FIG. 9A, the adjacent areas Aadjs and Aadje are omitted.

  As a result, the flash correction unit 80 corrects the image Im2 to the image Im2 'and corrects the image Im3 to the image Im3', as shown in FIG. 9B.

  Even when the area correction mode is set, as in the case where the screen unit switching mode is set, the flash correction unit 80 (effect processing unit 83) sets the entire corrected screen to a predetermined value or more in step S1410. A pseudo flash emission effect is added by adding the luminance signals.

  FIG. 9C shows a state in which a pseudo flash emission effect is added to the entire screen of the image Im2 'and the entire screen of the image Im3'. The images Im2 ′ and Im3 ′ are images Im2 ″ and Im3 ″ equivalent to a state in which the flash is emitted on the entire screen, as shown by hatching the entire screen.

  In the above description, a pseudo flash emission effect is added to the two images (Im2 ′ and Im3 ′) that are determined to have a high-luminance part by flash emission and the determination area is corrected. Only one of the images Im2 ′ and Im3 ′ (screen) may be used as an effect addition target screen, and a pseudo flash emission effect may be added, and the remaining images may be subjected only to correction.

  In FIG. 9, the image Im3 is corrected using the image Im2 to be an image Im3 ', but may be corrected using the image Im1. In this case, the buffer memory 4 needs to hold the image data of the previous two screens.

  In step S140 shown in FIG. 5, step S1401 is determined as to whether or not the screen unit switching mode is set, but may be determined as to whether or not the mode is the region correction mode.

  As described above, in the first embodiment, when the image data recorded in the recording / reproducing unit 11 is reproduced, the high-brightness portion is corrected by the flash emission, and a pseudo flash emission effect is added. Then, the image is displayed on a display device as an external device connected to the display unit 7 or the video camera.

  According to the first embodiment, the portion where the brightness of the image is discontinuous due to the flash emission is corrected, and a pseudo flash emission effect is added to the entire screen. The unnaturalness due to the discontinuity is eliminated, and it is possible to maintain a sense of presence when the flash is emitted.

  According to the first embodiment, it is possible to reduce the unnaturalness of movement caused by the same image data (screen) being continued in the screen unit switching mode. In addition, according to the first embodiment, unnaturalness due to partial correction of image data in the area correction mode can be reduced.

  According to the first embodiment, the image data is recorded in the recording / reproducing unit 11 in an uncorrected state, and even if the flash emission is erroneously detected during the reproduction of the image data, the recorded image data is recorded. Has no effect.

Second Embodiment
In the second embodiment shown in FIG. 10, the same parts as those in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted as appropriate. A video camera 102 as an image processing apparatus according to the second embodiment includes a flash detection unit 20 instead of the flash detection unit 2. The flash detection unit 20 includes a block average luminance calculation unit 201, a block average luminance holding unit 202, a line average luminance calculation unit 203, a line average luminance holding unit 204, a full screen average luminance calculation unit 205, a full screen average luminance holding unit 206, A difference calculation unit 207 and a flash determination unit 208 are included.

  The flash detection unit 20 detects flash emission according to control by the control unit 9 as follows. The block average luminance calculation unit 201 divides one line of effective pixels in the electrical signal supplied from the imaging unit 1 into a plurality of blocks, and calculates the average brightness for each block. The number of blocks is preferably a power of two. Specifically, the block average luminance calculation unit 201 adds the value of the G signal over the effective pixels of each block, and divides the value by the number of effective pixels to average the brightness of one block (block average luminance). Is calculated.

  The block average luminance calculated by the block average luminance calculation unit 201 is input to the block average luminance holding unit 202 and the line average luminance calculation unit 203. The line average luminance calculation unit 203 calculates the average (line average luminance) of the brightness of one line by adding the block average luminance in one line and dividing by the number of blocks in one line. The line average luminance calculated by the line average luminance calculation unit 203 is input to the line average luminance holding unit 204 and the full screen average luminance calculation unit 205. The line average luminance holding unit 204 temporarily holds the line average luminance calculated by the line average luminance calculating unit 203.

  The total screen average brightness calculation unit 205 adds the average line brightness sequentially input over the effective lines of one screen, and divides by the number of effective lines to average the brightness of the entire screen (average average screen brightness). Is calculated. The full screen average brightness holding unit 206 temporarily holds the full screen average brightness.

  The difference calculation unit 207 includes a block average luminance held by the block average luminance holding unit 202, a line average luminance held by the line average luminance holding unit 204, and a full screen held by the full screen average luminance holding unit 206. The average luminance is input. The difference calculation unit 207 sequentially calculates a difference between the block average luminance and the entire screen average luminance, and a difference between the line average luminance and the entire screen average luminance. The difference calculation unit 207 takes the absolute value of the difference and outputs it as a difference value.

  Each difference value calculated by the difference calculation unit 207 is input to the flash determination unit 208. The flash determination unit 208 determines whether or not the difference value between the line average luminance and the entire screen average luminance is equal to or greater than a set threshold (threshold 1). When the difference value between the line average luminance and the entire screen average luminance is equal to or greater than the threshold, the flash determination unit 208 is equal to or greater than the threshold (threshold 2) in which the difference value between each block average luminance and the entire screen average luminance is set. It is determined whether or not. The threshold value 1 and the threshold value 2 may be different values. When the threshold value 1 and the threshold value 2 are different from each other, it is preferable to make the value of the threshold value 2 larger than the threshold value 1.

  The flash determination unit 208 determines whether or not the state where the difference value for each line is equal to or greater than the threshold value 1 and the difference value for each block is equal to or greater than the threshold value 2 continues for a predetermined number of lines or more. The flash determination unit 208 detects that the brightness of the image has become discontinuous due to flash emission when it is determined that the state where the respective difference values are the threshold values 1 and 2 or more continues for a preset number of lines. .

  If the flash determination unit 208 determines that the flash is emitted, the difference value between the line average luminance and the entire screen average luminance, which is the start point of flash emission, is equal to or greater than the threshold 1, and each block average The data indicating the first line number for which the difference value between the luminance and the entire screen average luminance is equal to or greater than the threshold 2, the difference value between the line average luminance and the entire screen average luminance is equal to or greater than the threshold 1, and the respective block average luminances Data indicating the first line number that does not satisfy at least one of the conditions that the difference value with respect to the entire screen average luminance is not less than the threshold value 2 is output. The flash determination unit 208 also outputs a detection signal indicating that the flash has emitted light.

In the second embodiment, a step S30 ₂ shown in FIG. 11 the flash detection processing of step S30 shown in FIG. With reference to FIG. 11, details of the flash detection processing in step S30 _2. The following steps are executed by the flash detection unit 20 based on control by the control unit 9.

  In FIG. 11, in step S3201, the flash detection unit 20 sequentially executes the process of adding the pixel values of the effective pixels of each block in the effective line for each effective line. In step S3202, the flash detection unit 20 calculates the block average luminance by dividing the addition value of the pixel values of each block obtained in step S3201 by the number of effective pixels in the block, and holds the block average luminance.

  In step S3203, the flash detection unit 20 performs a difference calculation process between the block average luminance calculated in step S3202 and the average luminance of the entire screen one screen before. The difference calculation process in step S3203 is executed every time the block average luminance is calculated in step S3202. In step S3204, the flash detection unit 20 adds the block average luminance in one line, and in step S3205, calculates the line average luminance by dividing the added value of the block average luminance by the number of blocks. Hold.

  In step S3206, the flash detection unit 20 executes a process of sequentially adding line average luminance. In step S3207, the flash detection unit 20 executes a difference calculation process between the line average luminance calculated in step S3205 and the average average luminance of the previous screen. The difference calculation process in step S3207 is executed every time the line average luminance is calculated in step S3205.

  In step S3208, the flash detection unit 20 determines whether or not it is a normal state in which flash emission is not detected. If it is determined that the current state is the normal state (YES), whether or not the flash detection unit 20 has detected a line in which the difference value between the line average luminance and the entire screen average luminance is equal to or greater than the threshold (threshold 1) in step S3209. Determine whether.

  If it is determined that a line whose difference value between the line average luminance and the entire screen average luminance is equal to or greater than the threshold is detected (YES), the flash detection unit 20 shifts the process to step S3210, and the line average luminance and the entire screen average. If it is not determined that a line having a difference value with respect to the luminance has been detected as being greater than or equal to the threshold (NO), the flash detection unit 20 moves the process to step S3215. If a line having a difference value between the line average brightness and the full screen average brightness of a threshold value or more is not detected, the flash is not emitting light.

  If the flash detection unit 20 detects a line whose difference value between the line average luminance and the full screen average luminance is equal to or larger than the threshold value, the flash detection unit 20 further determines, in step S3210, each block average luminance and full screen average luminance in the line. It is determined whether or not the difference value is equal to or greater than a threshold value (threshold value 2). If the flash detection unit 20 determines that the difference value between each block average luminance and the entire screen average luminance is equal to or greater than the threshold (YES), the flash detection unit 20 shifts the processing to step S3211. At this time, the flash detection unit 2 stores detected line information indicating the number of detected lines from the first line of the frame.

  On the other hand, if it is not determined that the difference value between each block average luminance and the entire screen average luminance is equal to or greater than the threshold (NO), the flash detection unit 20 stores the detected line information if it is already stored. While erasing, the process proceeds to step S3215.

  In step S3211, the flash detection unit 20 determines that the difference value between the line average luminance and the full screen average luminance is equal to or greater than the threshold, and the difference value between each block average luminance and the full screen average luminance is equal to or greater than the threshold. It is determined whether or not the number of lines that satisfy one condition continues for a predetermined number of lines or more. If it is determined that the predetermined number of lines or more have been continued (YES), the flash detection unit 20 shifts the process to step S3214. If it is not determined that the predetermined number of lines or more has been continued (NO), the flash detection unit 20 performs the process. The process proceeds to step S3215.

  In step S3214, the flash detection unit 20 changes the flash detection state from a state in which flash emission is not detected to a state in which flash emission is detected. Further, the flash detection unit 20 supplies the stored detection line information to the flash correction unit 80 and erases the detection line information.

  On the other hand, if it is not determined in step S3208 that the current state is normal (NO), the flash detection unit 20 determines in step S3212 that the difference value between the line average luminance and the full screen average luminance is less than the threshold value, or Then, it is determined whether or not a line in which the difference value between each block average luminance and the entire screen average luminance is less than the threshold is detected. If it is determined that a line that does not satisfy at least one appears (YES), the flash detection unit 20 moves the process to step S3213. At this time, the flash detection unit 20 stores detection line information indicating the number of detected lines from the first line of the frame.

  If it is not determined that a line that does not satisfy at least one appears (NO), the flash detection unit 20 moves the process to step S3215.

  In step S3213, the flash detection unit 20 determines whether or not a state in which at least one of the above two conditions is not satisfied continues for a predetermined number of lines or more. If it is determined that the predetermined number of lines or more have been continued (YES), the flash detection unit 20 shifts the process to step S3214. If it is not determined that the predetermined number of lines or more has been continued (NO), the flash detection unit 20 performs the process. The process proceeds to step S3215.

  In step S3214, the flash detection unit 20 changes the flash detection state from the state where the flash emission is detected to the state where the flash emission is not detected. Further, the flash detection unit 20 supplies the stored detection line information to the flash correction unit 80 and erases the detection line information.

  In step S3215, the flash detection unit 20 determines whether the determination for all active lines is completed. If it is determined that the determination for all the effective lines is completed (YES), the flash detection unit 20 shifts the process to step S3216. If it is not determined that all the effective lines are completed (NO), the flash detection unit 20 Returns the process to step S3201, and repeats step S3201 and subsequent steps. In step S3216, the flash detection unit 20 calculates the total screen average brightness by dividing the added value of the line average brightness by the number of effective lines, and holds the total screen average brightness.

  According to the second embodiment, it is possible to improve the detection accuracy as to whether or not the flash has emitted, compared to the first embodiment. The reason why the detection accuracy can be improved will be described with reference to FIG.

  FIG. 12 shows an example of an image Imi in which a high-luminance area Arh that is partially rectangular in the screen is included. The hatched high luminance area Arh is assumed to have a white level, for example. The high brightness area Arh may not be rectangular. In the effective line Lne, the line average luminance is a relatively high value. The same applies to a range other than the effective line Lne in the vertical range where the high luminance area Arh exists. Therefore, in the first embodiment, there is a possibility that the flash is erroneously determined to emit light in the vertical range where the high luminance area Arh exists.

  FIG. 12 shows an example in which one line is divided into four blocks B1 to B4. The block average brightness calculated by the block average value calculation unit 201 is not so high in the block B1. The block average luminance is a high value in the blocks B2 and B3, and a smaller value in the block B4 than in the blocks B2 and B3. Even if it is determined in step S3209 in FIG. 11 that the difference value between the line average luminance and the entire screen average luminance is greater than or equal to the threshold value in the effective line Lne, in step S3210, at least block B1 has the block average luminance. Is not determined to be greater than or equal to the threshold value.

  Therefore, according to the second embodiment, it is possible to greatly reduce the possibility of erroneously determining that the flash has emitted even in the case of the image Imi including the partial high brightness area Arh as in the example of FIG. it can.

  According to the second embodiment, the same effect as that of the first embodiment by having the flash correction unit 80 is obtained, and whether or not the flash by providing the flash detection unit 20 instead of the flash detection unit 2 emits light. It is possible to improve the detection accuracy.

<Third Embodiment>
The configuration of the image processing apparatus of the third embodiment is the same as that of the video camera 101 of the first embodiment shown in FIG. 1 or the video camera 102 of the second embodiment shown in FIG. The operation in the third embodiment will be described with reference to FIGS. In FIG. 13, the same steps as those in FIG. FIG. 14 shows processing when the configuration of the image processing apparatus is the same as that of the video camera 101 of FIG.

  In FIG. 13, the control unit 9 determines whether or not there is an instruction to start shooting with the operation switch 12 in step S <b> 1. If it is determined that there is an instruction to start shooting (YES), the control unit 9 starts shooting by controlling the imaging unit 1 in accordance with conditions such as an aperture and a shutter speed in step S2. If it is not determined that there is an instruction to start shooting (NO), the control unit 9 returns the process to step S1 and repeats step S1.

  In step S <b> 300, the control unit 9 causes the flash detection process by the flash detection unit 2 and the flash correction process by the flash correction unit 80 to be executed. The control unit 9 causes the image signal processing unit 3 to execute image signal processing in step S4 in parallel with step S3. In step S5, the control unit 9 controls the output signal processing unit 6 to display the captured image on the display unit 7. The image displayed on the display unit 7 may be a captured image as it is, or an image that has been subjected to flash correction processing by the flash correction unit 80.

  In step S <b> 6, the control unit 9 determines whether or not there is an instruction to start recording by the operation switch 12. If it is determined that there is an instruction to start recording (YES), the control unit 9 performs the flash correction processing by the flash correction unit 80 and the compression data compressed by the compression / decompression processing unit 5 in step S7. Is recorded in the recording / reproducing unit 11. The image data subjected to the flash correction process by the flash correction unit 80 is temporarily held in the buffer memory 4 and is compressed by the compression / decompression processing unit 5. If it is not determined that there is an instruction to start recording (NO), the control unit 9 shifts the process to step S10.

  In step S8, the controller 9 determines whether or not there is an instruction to end recording. If it is determined that there is an instruction to end recording (YES), the control unit 9 stops recording in the recording / reproducing unit 11 in step S9. If it is not determined that there is an instruction to end recording (NO), the control unit 9 returns the process to step S7 and repeats step S7. Then, in step S10, the control unit 9 determines whether or not there has been an instruction to end shooting using the operation switch 12. If it is determined that there is an instruction to end shooting (YES), the control unit 9 ends the process. If it is not determined that there is an instruction to end shooting (NO), the control unit 9 returns the process to step S2 and repeats step S2 and subsequent steps.

  Details of the flash detection / correction processing in step S300 shown in FIG. 13 will be described with reference to FIG. The following steps are executed by the flash detection unit 2 and the flash correction unit 80 based on control by the control unit 9.

  In FIG. 14, in step S301, the flash detection unit 2 sequentially executes the process of adding the pixel values of the effective pixels in the effective line for each effective line. An effective line is a line in a period excluding a vertical blanking period, and an effective pixel is a pixel in a period excluding a horizontal blanking period. In step S302, the flash detection unit 2 calculates the line average luminance by dividing the addition value of the pixel values obtained in step S301 by the number of effective pixels, and holds the line average luminance.

  In step S303, the flash detection unit 2 executes a process of sequentially adding the line average luminance. In step S304, the flash detection unit 2 executes a difference calculation process between the line average luminance calculated in step S302 and the entire screen average luminance of the previous screen. The difference calculation process in step S304 is executed every time the line average luminance is calculated in step S302. In step S305, the flash detection unit 2 determines whether a line having a difference value equal to or greater than a threshold value is detected.

  The threshold value at this time should preferably be changed according to the condition of the subject, but an optimum value may be obtained by experiment and set. In order to detect a state in which the brightness is greatly changed by the light emission of another person's flash, the threshold value is preferably set to a value sufficiently larger than the dynamic range of the input G signal. The detection sensitivity of flash emission can be increased as the threshold value is decreased. Note that when the flash detection / correction process is started in step S300, it is preferable not to calculate the difference value because the average brightness of the entire screen of the previous one screen is not saved.

  If it is determined that a line having a difference value equal to or greater than the threshold value is detected (YES), the flash detection unit 2 shifts the process to step S306. At this time, the flash detection unit 2 stores detected line information indicating the number of detected lines from the first line of the frame. The detected line information to be stored may be only the first line when the lines are continuously detected. On the other hand, if it is not determined that a line having a difference value equal to or greater than the threshold value has been detected (NO), the flash detection unit 2 deletes the detected line information if already stored, and the process proceeds to step S314. Transition. If no line with a difference value equal to or greater than the threshold value is detected, the flash is not emitting light.

  In step S306, the flash detection unit 2 determines whether or not the state where the difference value is equal to or greater than the threshold value continues for a predetermined number of lines or more. If it is determined that the predetermined number of lines or more have been continued (YES), the flash detecting unit 2 shifts the process to step S307. If it is not determined that the predetermined number of lines or more has been continued (NO), the flash detecting unit 2 The process proceeds to step S314.

  By providing step S306, erroneous detection can be prevented even if the threshold value in step S305 is reduced. The number of lines in step S306 may be set by obtaining an optimum value by experiment. Even if a line having a difference value equal to or greater than the threshold value is detected, if the state where the difference value is equal to or greater than the threshold value has not continued for a predetermined number of lines, it can be determined that the flash is not emitting light.

  If it is determined in step S306 that the predetermined number of lines or more have been continued, this means that the flash has been emitted. In step S307, the flash detection unit 2 switches from the detection signal “0” indicating that the flash is not emitting to the detection signal “1” indicating that the flash is emitted, The stored detection line information is supplied to the flash correction unit 80 and the detection line information is erased. In step S308, the flash detection unit 2 and the flash correction unit 80 determine whether or not the set mode is the region correction mode. If it is determined that the region correction mode is set (YES), the flash correction unit 80 shifts the processing to step S309. If it is not determined that the area correction mode is set (NO), the flash detection unit 2 shifts the process to step S316.

  In step S309, the flash correction unit 80 starts replacement with the previous line indicated by the supplied detection line information. In step S310, the flash detection unit 2 determines whether a line having a difference value less than the threshold value is detected. If it is determined that a line having a difference value less than the threshold value is detected (YES), the flash detection unit 2 shifts the process to step S311. At this time, the flash detection unit 2 stores detected line information indicating the number of detected lines from the first line of the frame. The detected line information to be stored may be only the first line when the lines are continuously detected. On the other hand, if it is not determined that a line having a difference value less than the threshold value has been detected (NO), the flash detection unit 2 deletes the stored detection line information and shifts the process to step S314.

  In step S311, the flash detection unit 2 determines whether or not the state where the difference value is less than the threshold value continues for a predetermined number of lines or more. If it is determined that the predetermined number of lines or more have been continued (YES), the flash detection unit 2 shifts the process to step S312. If it is not determined that the predetermined number of lines or more has been continued (NO), the flash detection unit 2 The process proceeds to step S314.

  By providing step S311, erroneous detection can be prevented even if the threshold value in step S310 is reduced. The threshold value in step S305 and the threshold value in step S310 are the same value. The number of lines in step S311 may be set by obtaining an optimum value by experiment. The predetermined number of lines in step S306 and the predetermined number of lines in step S311 may be the same. Even if a line having a difference value less than the threshold value is detected, if the state where the difference value is less than the threshold value does not continue for a predetermined number of lines or more, it can be determined that the flash emission has not ended.

  In step S312, the flash detection unit 2 switches the detection signal “1” to the detection signal “0” and supplies the switched detection signal and the stored detection line information to the flash correction unit 80. Delete detection line information. In step S313, the flash correction unit 80 ends the replacement with the previous line indicated by the supplied detection line information, and the flash detection unit 2 moves the process to step S314. In step S314, the flash detection unit 2 determines whether the determination for all active lines has been completed. If it is determined that the determination for all the effective lines is completed (YES), the flash detection unit 2 shifts the process to step S315, and if it is not determined that all the effective lines are completed (NO), the flash detection unit 2 Returns the process to step S305 and repeats step S305 and subsequent steps.

  In step S315, the flash detection unit 2 calculates the total screen average brightness by dividing the added value of the line average brightness by the number of effective lines, holds the total screen average brightness, and proceeds to step S320. In step S320, the flash correction unit 80 adds a pseudo flash emission effect by adding a luminance signal of a predetermined value or more to the entire corrected screen, and proceeds to step S5 in FIG.

  On the other hand, if it is not determined in step S308 that the area correction mode is set (that is, it is determined that the screen unit switching mode is set), the flash detection unit 2 ends the determination for all effective lines in step S316. Determine whether or not. If it is not determined that the determination for all active lines is completed (NO), the flash detection unit 2 repeats the process of step S316. If it is determined that the determination for all the effective lines is completed (YES), in step S317, the flash detection unit 2 calculates the total screen average luminance by dividing the added value of the line average luminance by the number of effective lines. Holds the full screen average brightness.

  In step S318, the flash correction unit 80 determines whether or not the screen has shifted to the next screen. Information on whether or not the screen has shifted to the next screen may be supplied from the flash detection unit 2 or the control unit 9 to the flash correction unit 80. If it is not determined that the screen has shifted to the next screen (NO), the flash correction unit 80 repeats the process of step S318. If it is determined that the screen has shifted to the next screen (YES), in step S319, the flash correction unit 80 replaces the current screen in which flash emission has been detected with a normal screen in the past, and then proceeds to step S320. Let

  Similarly, in the screen unit switching mode, the flash correction unit 80 adds a pseudo flash emission effect by adding a luminance signal of a predetermined value or more to the entire corrected screen in step S320. Then, the process proceeds to step S5 in FIG.

  In FIG. 14, the process of step S320 is provided after the steps S315 and S319, but the position where the step S320 is provided is not limited to the position shown in FIG.

  In the third embodiment, the flash correction unit 80 corrects at least a line in a region that is bright due to flash emission, and adds a pseudo flash emission effect, as in FIGS. The image data subjected to the correction processing can be recorded in the recording / reproducing unit 11. Therefore, in the third embodiment, as in the first embodiment, it is necessary to cause the recording / reproducing unit 11 to record the detection line information indicating the start line and the end line of the flash emission detected by the flash detection unit 2 as metadata. Absent.

  As described above, in step S5, the image data may be displayed on the display unit 7 without being corrected, or the image data corrected by the flash correction unit 80 may be displayed on the display unit 7. Even if the brightness of the flash is partially increased, if the image data is displayed on the display unit 7 without correction, the image data to be displayed on the display unit 7 is not delayed. If the image data corrected by the flash correction unit 80 is displayed on the display unit 7, the corrected image data can be confirmed without reproducing the image data recorded in the recording / reproducing unit 11 later.

  FIG. 14 shows processing when the configuration of the image processing apparatus is the same as that of the video camera 101 of FIG. 1 having the flash detection unit 2, but the configuration of the image processing apparatus is changed to that of the flash detection unit 20. The processing in the case of the same configuration as the video camera 102 of FIG. That is, also in the second embodiment, the image data corrected by the flash correction unit 80 can be recorded in the recording / reproducing unit 11.

  The present invention is not limited to the embodiments described above, and various modifications can be made without departing from the scope of the present invention. In each embodiment, a video camera is taken as an example, but the image processing apparatus and method of the present invention can be used in any electronic device that handles image data. For example, the image processing apparatus according to each embodiment is mounted on an information processing apparatus such as an image display apparatus or a computer, and image data in which part of the screen has high brightness is corrected inside the image display apparatus or information processing apparatus. You may do it.

  Moreover, in each embodiment, although the structure which correct | amends the video data which completed the process in the image signal processing part 3 was shown, you may correct | amend the video data before processing by the image signal processing part 3. FIG. In each embodiment, the configuration using the buffer memory 4 provided independently of the image signal processing unit 3 and the compression / decompression processing unit 5 has been described. However, the buffer used in the image signal processing unit 3 and the compression / decompression processing unit 5 is described. A memory may be used, and the circuit (block) configuration can be changed as appropriate.

  A configuration equivalent to the image processing apparatus and method of the present invention can be realized by a computer program (image processing program). The image processing program may be provided by being recorded on a recording medium, or the image processing program may be distributed through a communication line such as the Internet. An image processing program recorded on a recording medium or an image processing program distributed via a communication line may be stored in an image processing apparatus to execute the above-described image processing method.

DESCRIPTION OF SYMBOLS 1 Image pick-up part 1s CMOS sensor 2,20 Flash detection part 3 Image signal processing part 4 Buffer memory (holding part)
DESCRIPTION OF SYMBOLS 5 Compression / decompression process part 6 Output signal process part 7 Display part 9 Control part 10 Memory 11 Recording / reproducing part 12 Operation switch 21,203 Line average brightness | luminance calculation part 23,205 Whole screen average brightness | luminance calculation part 25,207 Difference calculation part 26, 208 Flash determination unit 80 Flash correction unit 81 Correction control unit 82 Correction processing unit 83 Effect processing unit 101, 102 Video camera 201 Block average luminance calculation unit

Claims (7)

A flash detection unit that detects whether or not a line that is bright due to flash emission is included in a screen composed of image data;
A holding unit for holding a past screen of the image data;
When the flash detection unit detects that a line with high luminance is included, at least the line with high luminance is replaced with the line of the past screen held in the holding unit. A correction processing unit for correcting image data by
An effect processing unit that sets at least one of the screens corrected by the correction processing unit as an effect addition target screen and adds a pseudo flash emission effect to image data constituting the effect addition target screen;
A recording unit for recording image data to which a pseudo flash emission effect is added by the effect processing unit;
An image processing apparatus comprising:   2. The image according to claim 1, wherein the effect processing unit adds a pseudo flash emission effect by adding a fixed value of a luminance signal to the image data constituting the effect addition target screen. Processing equipment.   The effect processing unit adds a pseudo flash emission effect by multiplying a luminance signal and a color difference signal of image data constituting the effect addition target screen by individually set gains. The image processing apparatus according to claim 1. The flash detection unit
A line average luminance calculation unit for calculating the line average luminance of each line in the detection target screen;
A full screen for calculating a total screen average brightness of a past screen, which is a screen at least one screen before the detection target screen, based on the line average brightness of each line in one screen calculated by the line average brightness calculation unit. An average luminance calculator;
A flash that determines whether or not a line in the detection target screen has high luminance due to flash emission by comparing the line average luminance of each line in the detection target screen with the average luminance of the entire screen. A determination unit;
The image processing apparatus according to claim 1, further comprising: The flash detection unit
A block average luminance calculating unit for calculating a block average luminance of each block obtained by dividing a line on the detection target screen into a plurality of blocks;
A line average luminance calculating unit that calculates line average luminance of each line in the detection target screen based on the block average luminance in one line calculated by the block average luminance calculating unit;
A full-screen average luminance calculating unit that calculates the full-screen average luminance of the past screen based on the line average luminance of each line in one screen calculated by the line-average luminance calculating unit;
A difference calculating unit for calculating a difference between the block average luminance and the full screen average luminance, and a difference between the line average luminance and the full screen average luminance, respectively;
The condition that the difference between the line average luminance and the full screen average luminance is equal to or greater than a first threshold and the difference between the block average luminance and the full screen average luminance is equal to or greater than a second threshold continues for a predetermined line or more. A flash determination unit that determines that the brightness is high due to flash emission,
The image processing apparatus according to claim 1, further comprising: The flash detection unit detects a flash start line and a flash end line that are the first line that has become bright due to flash emission, and the last line,
The correction processing unit is located above the flash start line, is adjacent to the flash start line, is located below the flash end line, and is adjacent to the flash end line. 2 adjacent regions are set, and in the first adjacent region, the image data of the detection target screen is sequentially reduced as the first line advances from the first line to the last line of the first adjacent region. Both image data are mixed so as to sequentially increase the image data of the screen of the previous screen, and in the second adjacent area, the past screen of the previous screen is progressed from the first line to the last line of the second adjacent area. 5. The image data is mixed in such a manner that the image data is sequentially decreased and the image data on the detection target screen is sequentially increased. Or the image processing apparatus of 5. Detect whether the screen composed of image data contains lines that are bright due to flash emission,
Holding a past screen of the image data in a holding unit;
When it is detected that a line with high luminance is included, the image data is corrected by replacing at least the line with high luminance with the line of the past screen held in the holding unit. And
At least one of the corrected screens is set as an effect addition target screen, and a pseudo flash emission effect is added to the image data constituting the effect addition target screen,
An image processing method, wherein image data to which a pseudo flash emission effect is added is recorded in a recording unit.

Images