JP5875611B2 - Image processing apparatus and image processing apparatus control method - Google Patents

Description

  The present invention relates to an image processing apparatus that gives a film-like effect to digital image data.

  In recent years, as one of the image expressions of a digital camera, there has been proposed a synthesis method for adding granular noise (hereinafter referred to as grain noise) to a digital image in order to produce a grainy effect by film shooting on the captured image.

  In Patent Document 1, grain pattern data, which is the source of grain noise, is calculated from a film, and grain noise is added by arranging a plurality of noise data cut out at random positions.

Japanese Patent Laid-Open No. 11-085955

However, the phenomenon that occurs in an image when reproduced over the film projector, not only the grain noise in accordance with the whole image as described above, has not been able to grant them as video effect simultaneously. The present invention has been made in view of the above-described problems, and it is an object of the present invention to provide an image processing apparatus capable of giving a plurality of different video effects to an image.

In order to solve the above problems, an image processing apparatus of the present invention includes an acquisition unit that acquires an image of a plurality of frames, the image each of the plurality of frames, switching to perform out Ri cut at positions determined randomly And a masking unit for superimposing a mask image on the clipped image. The image processing apparatus according to the present invention includes an acquisition unit that captures an image of a subject and sequentially acquires a plurality of frames of images, and a correction that randomly changes luminance input / output characteristics with respect to the plurality of frames of images. And a recording unit that records a plurality of frames of images corrected by the luminance correcting unit as a moving image on a recording medium. The image processing apparatus of the present invention includes an acquisition unit that acquires a plurality of frames of images, a distortion processing unit that performs a process of distorting the images of the plurality of frames, and a plurality of frames processed by the distortion processing unit. Noise processing means for performing processing for adding noise to an image, blur addition processing means for performing processing for adding blur to an image of a plurality of frames processed and output by the noise processing means, and processing by the blur addition processing means Recording means for recording the plurality of frames of images as a moving image on a recording medium. The image processing method of the present invention includes an acquisition step of acquiring a plurality of frames of images, a step of cutting out each of the plurality of frames of images at a position determined at random,
And a mask step for superimposing a mask image having a set size on the clipped image. In addition, the control method of the imaging apparatus of the present invention includes an acquisition step of capturing a subject and sequentially acquiring a plurality of frames of images, and changing luminance input / output characteristics at random with respect to the plurality of frames of images. A brightness correction step for performing correction, and a recording step for recording a plurality of frames of images corrected in the brightness correction step on a recording medium as a moving image. The image processing method of the present invention includes an acquisition step of acquiring a plurality of frames of images, a distortion processing step of performing a process of distorting the images of the plurality of frames, and a plurality of frames processed in the distortion processing step. A noise processing step for performing processing for adding noise to an image, a blur addition processing step for performing processing for adding blur to a plurality of frames of images that are processed and output in the noise processing step, and processing in the blur addition processing step A recording step of recording the plurality of frames of images as a moving image on a recording medium.

As described above, according to the present invention, it is possible to artificially give an image an image effect resulting from a plurality of phenomena that occur when an input image is reproduced by applying a film to a projector.

1 is a block diagram illustrating an example of an image processing apparatus according to a first embodiment. A block diagram showing an example of an image processing device in a 2nd embodiment The figure which shows the noise data of the grain noise in 1st Embodiment The figure which shows the structure of the synthesized image with the grain noise in 1st Embodiment The figure which shows the noise data of the scratch flaw in 2nd Embodiment The figure which shows the structure of a synthesized image with the noise data of a scratch flaw in 2nd Embodiment Flowchart for determining the cut-out position of noise data of grain noise in the first embodiment The figure which shows the cutout position of the noise data of the grain noise in 1st Embodiment. The table | surface which shows the position after correction | amendment of the cutting-out position of the noise data of the grain noise in 1st Embodiment The figure which shows the duplication degree by the difference in the correction position of the noise data of the grain noise in 1st Embodiment. Flowchart for determining the position to paste noise data for scratches in the second embodiment The figure which shows the time change of the necessity or non-necessity of the addition of a scratch in 2nd Embodiment. Block diagram of an image processing apparatus according to the third embodiment The figure which shows the frame memory and image clipping in 3rd Embodiment The figure which shows the process which conceals the noise by clipping in 3rd Embodiment Flowchart of vertical shake processing in the third embodiment The figure which shows the several different kind of blur used for the up-and-down shake in 3rd Embodiment, and its combination A block diagram showing an example of an image processing device in a 4th embodiment The figure which shows the correction | amendment process of the luminance signal for implement | achieving blinking in 4th Embodiment Flowchart of luminance signal correction processing in the fourth embodiment A block diagram showing an example of an image processing device in a 5th embodiment The flowchart figure of the process which accumulates the effect of the some film tone in 5th Embodiment A block diagram showing an example of an image processing device in a 6th embodiment The figure explaining the process from recording to reproduction | regeneration in the video recording which used the film in 6th Embodiment as the recording medium In each process shown in FIG. 24, a table on which events that affect the screening results, effects by the events, and means for realizing the effects by image processing in a pseudo manner are provided. Flowchart of processing for overlapping the effects of a plurality of film tones in the sixth embodiment The block diagram which shows an example of the image processing apparatus in other embodiment

  Hereinafter, preferred embodiments of the present invention will be described.

<First Embodiment>
In the present embodiment, an image processing apparatus capable of adding grain noise, which is granular noise, to an image in order to give a granular feeling as a film-like noise effect is shown.

  FIG. 1 is a block diagram of a digital video camera as an image processing apparatus according to the first embodiment. The image sensor 100 photoelectrically converts the incident light that has been imaged. The photoelectrically converted signal is input to the camera signal processing unit 101. The camera signal processing unit 101 performs various kinds of image processing on the photoelectrically converted signal, converts it into a video signal, and outputs it. The output video signal is encoded by the encoder unit 111 in a predetermined recording format and recorded on the recording medium 112.

  Next, signal processing in the camera signal processing unit 101 will be described. The signal input to the camera signal processing unit 101 is first subjected to various signal processing by the image processing unit 104. Various signal processing includes white balance processing, gamma processing, color space conversion processing, color gain processing, color balance processing, and the like performed on image data during normal shooting. The processed signal is stored in the first frame memory 108 as a captured image.

  The two-dimensional 109 is a memory that stores two-dimensional noise data 301 as grain noise. The noise data 301 is read from the memory 109. The cutout processing unit 106 cuts out noise data 302 having a predetermined position and size from the memory data 301. The cut out noise data 302 is stored in the second frame memory 110. The noise data 302 read from the second frame memory 110 is resized by the enlargement processing unit 107 into noise data 402 having a size necessary for synthesis with the captured image.

  The synthesis processing unit 105 reads the captured image 401 stored in the first frame memory 108 and the noise data 402 processed by the enlargement processing unit 107 at a predetermined timing, performs synthesis, and outputs a synthesized image. The composition processing unit 105 can change the composition ratio of the captured image 401 and the noise data 402. The intensity of added grain noise can be changed by changing the composition ratio. A method for generating and adding grain noise will be described in detail later.

  The signal generator 103 is a signal generator (SG) that generates a signal for controlling the drive timing of the image sensor 100. The generated signal is supplied to the image sensor 100.

  A system controller 102 controls a camera block including the image sensor 100 and the camera signal processing unit 101. A signal accumulation period and readout timing are instructed to the image sensor 100. For the image processing unit 104, parameters necessary for image quality setting are set by various signal processing. For the synthesis processing unit 105, a frame memory to be synthesized is designated, a synthesis ratio is designated, and a synthesis timing is instructed. In order to generate grain noise, the cutout processing unit 106 is instructed to read out from the memory 109, the cutout position and size, and the timing to write out the cutout data to the second frame memory 110. I do. The enlargement processing unit 107 is instructed to read data from the second frame memory 110 and to instruct the size at the time of resizing. The signal generator 103 is instructed to drive timing for driving the image sensor 100.

  Here, with reference to FIGS. 3 and 4, the generation method and the addition method of the grain noise used when adding the grain noise that gives the film-like grain effect to the image data, which is a characteristic process of the present embodiment, are described in detail. Describe. In the method of arranging a plurality of granular pattern data as in Patent Document 1, in order to arrange the pattern data in synchronization with the update period of the moving image, that is, the frame rate, a high-speed that performs different arrangements each time within the frame rate period. A simple processing system is required. Further, in the method of selecting and adding a plurality of grain noise patterns, a plurality of memory areas for storing the grain noise are required.

  Therefore, in this embodiment, grain noise is previously stored as two-dimensional data, noise data corresponding to the entire frame of an image input at an arbitrary position is extracted, and the extracted noise data is combined with a moving image. . At this time, by synchronizing the cut-out timing with the frame rate of the moving image and changing the cut-out position every frame, it becomes possible to synthesize spatially and temporally random grain noise into the moving image. .

  FIG. 3 is a diagram showing the relationship between two-dimensional noise data (grain noise data 301) and cut-out noise data 302 cut out therefrom. FIG. 4 is a diagram showing the concept of combining the cut-out noise data 302 and the captured image. In the present embodiment, the grain noise data 301 is a pixel group in which a minimum granularity unit is one pixel and a random number according to a Gaussian distribution is recorded for each pixel as a pixel value. Here, the random numbers are not limited to those according to the Gaussian distribution, and many types of random numbers such as uniform random numbers and exponential random numbers are conceivable, and the types of random numbers are not limited. Various forms such as image data or a data string can be considered as the form of actual data as noise data, and the form is not limited to any of these forms. Further, the minimum granular unit is not limited to one pixel, but of course, the smaller the unit, the higher the randomness.

  For the grain noise data 301, the cutout start positions X and Y are randomly determined by using dynamic values that can be acquired within the camera, such as processing time, camera posture information, and position information. Data of a predetermined size is cut out from the grain noise data 301 as cut-out noise data 302 from the determined start position, and is combined with the picked-up image as shown in FIG. 4 to add grain noise to the picked-up image.

  Here, when the cutout start position is randomly changed each time, if the previous cutout position and the current cutout position are close to each other or only shifted in the horizontal or vertical direction, a two-dimensional pattern is obtained as a synthesis result. Appears to move.

  Therefore, in the present embodiment, the current cutout position is set so that the amount of movement from the previous cutout position to the current cutout position does not fall within a predetermined range.

  With reference to FIG. 7, the clipping control of the noise data 301 instructed by the system controller 102 in this embodiment to the clipping processing unit 106 will be described. This control is performed in a shooting mode in which a film-like noise effect is added to a shot image. Alternatively, the moving images stored in the memory such as the recording medium 212 are continuously read out and performed as post processing.

  In step S701, candidate values to be extracted from this time are instructed from the noise data stored in the memory 109 using dynamic values as described above. In order to synthesize the cut-out data as grain noise, it is determined at random each time and is set to a position different from the previous cut-out position. When the current cutout position to be instructed is determined, the process proceeds to step S702.

  In step S702, the current frame rate information of the image sensor 100 is acquired based on the drive timing signal instructed to the signal generator 103. If information is acquired, it will progress to step S703.

  In step S703, a determination reference value for determining how far the cutout position instructed this time is from the previous cutout position stored in the temporary memory (not shown) is calculated. Since the grain noise to be synthesized in synchronism with the frame rate is updated, the grain noise update period becomes longer as the frame rate becomes smaller, that is, the captured image update period becomes longer, and it continues to be seen. Therefore, if the previous cutout position and the current cutout position are close to each other, the synthesized grain noise appears to move. In view of this, the reference (predetermined value) for determining how close the current cutout position is to the previous cutout position is changed according to the frame rate information acquired in step S702. For example, at a frame rate of 60 fps (first frame rate), when the current position is within 10 pixels (first predetermined value) around the previous position, it is determined that the position is close. On the other hand, at 30 fps (second frame rate), the determination criterion is determined so as to determine that the current position is within 20 pixels (second predetermined value) in the vicinity and that the position is close to the predetermined value or less. When the determination criterion is determined, the process proceeds to step S704.

  In step S704, a distance between cutout positions is calculated that indicates how much the current cutout position calculated in step S701 has changed from the previous cutout position of the noise data on the grain noise data. If it calculates, it will progress to step S705.

  In step S705, the distance from the previous cutout position calculated in step S704 to the cutout position instructed this time is compared with the determination criterion calculated in step S703. If it is determined that the distance is greater than the criterion, that is, the cutout position has changed significantly from the previous time to this time, the process proceeds to step S707. On the other hand, if it is determined that the distance is smaller than the criterion, that is, the cutout position has changed slightly from the previous time to this time, the process proceeds to step S706.

  In step S706, correction is performed on the position information of the cutout position instructed this time calculated in step S701. A method for correcting the position information of the cutout position will be described later. When the position information of the cutout position is corrected, the process proceeds to step S707.

  In step S707, the position information of the cutout position corrected in step S706 is stored in a temporary memory so as to be used for the next cutout position control, and the process proceeds to step S708.

  In step S708, the cutout position corrected in step S706 is instructed to the cutout processing unit 106, and the process ends.

  Next, the cutout position correction processing in step S706 will be described. 8, 9, and 10 are diagrams illustrating correction conditions and correction positions when correcting the cutout position from the noise data 301 stored in the memory 103. It is assumed that the width and height of the noise data 301 are W pixels and H pixels, respectively (W ≧ 4, H ≧ 4). The noise data is cut out in units of pixels, and the upper left is the origin (0, 0) of the cutting position. The size of the cut-out noise data 302 to be cut out is a width (W / 2) pixel and a height (H / 2) pixel.

  For example, as shown in FIG. 3, when the current cutout position is (X, Y), the cutout start position (X, Y) for cutting out the cutout noise data of the above size is (0 ≦ X <W / An area in the range of 2, 0 ≦ Y <Y / 2) can be specified (FIG. 8). Further, the region where the start position of cutout can be specified is divided into four regions, regions 1, 2, 3 and 4 shown in FIGS. 8 (a), (b), (c) and (d). Region 1 is a region including the cutout position (0 ≦ X <W / 4, 0 ≦ Y <H / 4), and region 2 is the cutout position (W / 4 ≦ X <W / 2, 0 ≦ Y <). H / 4). Further, the region 3 is (0 ≦ X <W / 4, H / 4 ≦ Y <H / 2), and the region 4 is (W / 4 ≦ X <W / 2, H / 4 ≦ Y <H / 2). It is an area including the cutout position in each range.

  When correcting the cutout position, the noise data cut out at the previous cutout position and the cutout position instructed this time have a smaller area of overlapping data, and if the position is farther away, the randomness at the time of synthesis is higher. It is preferable because it improves. Therefore, in the correction control, it is determined where the current cutout position is in the areas 1 to 4, and the cutout position is corrected on the diagonal area in accordance with the location. At this time, as shown in the table of FIG. 9, the cutout position (X, Y) is corrected to the corrected cutout position (X ′, Y ′).

  For example, if the current cut-out position is in area 1 and the corrected position is set to area 4 (FIG. 10 1003), compared to the case of moving to area 2 and area 3 (FIG. 10 1001, 1002). The area where the cut out noise data overlaps is reduced. As a result, the movement of the noise data is less noticeable, and the temporal randomness of the grain noise is improved.

  As described above, in the first embodiment, an arbitrary range is randomly extracted from the two-dimensional noise data every time the captured image is updated, and this is combined with the captured image. Thereby, it is possible to generate an image to which granular noise is added even if it is a moving image. Further, even if the randomly determined cut-out position is close to the previous range within the predetermined range, the randomness of the noise can be ensured by correcting the position outside the predetermined range. In this embodiment, compared to patent literature, noise data can be generated simply by cutting out data of a predetermined size from already read data, and the time required for generation can be reduced.

  Furthermore, if the current cut position of the noise data is nearer than the previous cut position, the cut noise data moves by changing the current cut position. Noise randomness is improved.

  In the present embodiment, the size of the cut-out noise data with respect to the grain noise data 301 (width W, height H) is set to width (W / 2) pixels and height (H / 2) pixels. However, the size of the cut-out noise data with respect to the grain noise data is not limited to this, and may be larger or smaller than this embodiment as long as it does not exceed the size of the grain noise data. However, if the size of the cut-out noise data is increased, the randomness is easily lost. If the size is small, the grain noise data 301 is too large for the required size of the cut-out noise data. End up. Therefore, in the present embodiment, the ratio of the size of the noise data 301 and the cut-out noise data is set to about 4: 1, and further, the processing of the present embodiment for controlling the start position not to be close is included, thereby ensuring the randomness of noise. And saving memory capacity are more favorably realized.

  In this embodiment, the area where the cutout start position can be divided is divided into four. However, the allocation of the number of divisions and the divided areas is not limited to this.

  Further, in the present embodiment, it is determined whether the distance between the current and previous cutout positions is equal to or smaller than a predetermined value, and when the distance is equal to or smaller than the predetermined value, correction for moving the region of the cutout start position as described above is performed. Went. However, the correction method is not limited to this, and the coordinate may be simply moved beyond the distance at which the distance between the current and previous cutout positions is greater than a predetermined value. That is, when the predetermined value is 10 pixels around at a frame rate of 60 fps, the position (coordinates) may be changed so that it becomes 11 pixels or more around. In the first place, when designating the start position of this time from the beginning, coordinates within 10 pixels in the vicinity may be removed from the candidates and designated at random.

  Further, in this embodiment, it is determined whether the distance between the current and previous cutout positions is equal to or less than a predetermined value, and when the distance is equal to or less than the predetermined value, the correction for moving the cutout start position area as described above is performed. Went. However, the method for determining the cutout start position is not limited to this, and the area to which the current and previous cutout positions belong is determined (whether it belongs to any of the areas 1, 2, 3, and 4 in FIG. 8), and the same area. May be corrected so as to be moved to a different area. At this time, if the movement of the area is a diagonal area, the area where the previous and current cutout areas overlap is smaller as in the present embodiment, which is preferable. Furthermore, when this judgment method is combined with the method for judging whether or not the distance between the current and previous cutout positions is equal to or less than a predetermined value, it is effective even when the start position is near the boundary of the region. Judgment is possible.

<Second Embodiment>
In the present embodiment, an image processing apparatus capable of adding a scratch to the image, which is vertical line noise, in order to give an effect of scratching the film as a film-like effect is shown. 5 and 6 explain the extraction and synthesis of noise data for scratches in the present embodiment. FIG. 5 is a diagram showing a relationship between noise data composed of a plurality of patterns of scratches and noise data cut out therefrom. Scratch flaw noise data 501 has one pixel in the horizontal direction as a minimum unit, scratch scratches are stored in the vertical direction, and the intensity is determined by a random number. In the vertical direction, the intensity changes with a plurality of pixels as a minimum unit. As a result, the density and thickness of the scratches change in the vertical direction, thereby expressing the “blur” of the scratches. Various random numbers can be considered, including a Gaussian distribution, but it does not depend on the type.

  FIG. 6 is a diagram showing the concept of combining the cut scratch and the captured image. In the present embodiment, the cut-out noise data 502 is cut out from the scratch flaw noise data 501 as shown in FIG. 5 and resized to a predetermined image size to generate the pasted noise data 602. Then, the pasting position of pasting noise data 602 is determined and pasted based on the pasting position of pasting noise data 602 and the pasting time at that position.

  FIG. 2 is a block diagram of a digital video camera as an image processing apparatus according to the second embodiment. The image sensor 200 photoelectrically converts the incident light that has been imaged. The photoelectrically converted signal is input to the camera signal processing unit 201. The camera signal processing unit 201 performs various kinds of image processing on the photoelectrically converted signal, converts it into a video signal, and outputs it. The output video signal is encoded in a predetermined recording format by the encoder unit 211 and recorded on the recording medium 212.

  Next, signal processing in the camera signal processing unit 201 will be described. The signal input to the camera signal processing unit 201 is first subjected to various signal processing in the image processing unit 204 as in the first embodiment. The processed signal is stored in the frame memory 209 as a captured image.

  A memory 210 stores scratch scratch noise data 501 including a plurality of patterns. The noise data 501 is read from the memory 210. The cut-out processing unit 206 cuts out the noise data 502 of scratches having a designated position and size on the noise data 501. The extracted noise data 502 is resized by the enlargement processing unit 207 into noise data 602 having a size necessary for synthesis with the captured image 601. The pasting processing unit 208 designates a pasting position for synthesizing the noise data 602 at an arbitrary position independent of the cutout position.

  The synthesis processing unit 205 reads the captured image 601 stored in the frame memory 209 and the noise data 602 processed by the pasting processing unit 208 at a predetermined timing, performs synthesis, and outputs a synthesized image.

  A signal generator (SG) 203 is a signal generator that generates a signal for controlling the drive timing of the image sensor 200. The generated signal is supplied to the image sensor 200 and the system controller 202.

  A system controller 202 controls a camera block including the image sensor 200 and the camera signal processing unit 201. A signal accumulation period and readout timing are instructed to the image sensor 200. For the image processing unit 204, parameters necessary for image quality setting are set by various signal processing. For the synthesis processing unit 205, the frame memory to be synthesized is designated, the synthesis ratio is designated, and the synthesis timing is instructed. The cutout processing unit 206 is instructed to read data from the memory 210 and instruct the cutout position and size in order to generate a scratch. The enlargement processing unit 207 is instructed to specify the size at the time of resizing. The paste processing unit 208 is instructed of the paste position at the time of synthesis. The signal generator 203 is instructed to drive timing for driving the image sensor 200.

  Next, the pasting control of the pasting noise data 602 instructed by the system controller 202 in the present embodiment to the pasting processing unit 208 will be described with reference to FIGS. This control is performed in a shooting mode in which image processing is performed as if a scratch is added to the shot image. Alternatively, the moving images stored in the memory such as the recording medium 212 are continuously read out and performed as post processing. This processing is started in conjunction with the vertical synchronization signal, and a series of processing is repeated by the synchronization signal.

  In step S1101, the value of the superimposition counter (first detection means), which is a counter that measures the time during which the scratches stored in the temporary memory (not shown) are superimposed on the captured image, is set to the superimposition target time (first detection means). It is determined whether or not the first predetermined time has been reached. If the target time has not been reached, the process proceeds to step S1108. On the other hand, if it has reached, the process proceeds to step S1102. Here, the target time for superimposition is a time for determining how long the same scratches can be continuously displayed on the captured image, and may be determined according to various conditions or may be determined by the user. .

  In step S1102, the superposition counter that has reached the target time is cleared to the initial state, and the process proceeds to step S1103.

  In step S1103, it is acquired whether or not this time scratch scratches extracted from the noise data stored in the memory 210 are to be superimposed. For example, the presence / absence information may be a value randomly obtained from random number generation means in order to randomly superimpose scratches. When the presence / absence information is acquired, the process proceeds to step S1104.

  In step S1104, it is determined whether the presence / absence information obtained in step S1103 is superimposed. The determination result is stored in a temporary memory. Presence / absence information may be defined as a binary value. For example, if it is a value obtained from random number generation means, superimposition is present if the value is greater than a predetermined threshold, and conversely if there is less than the threshold. And decide. If it looks as a time change, the process of step S1104 is executed, and as shown in FIG. 12, periods with and without superimposition occur randomly for each predetermined frame. In the present embodiment, the presence / absence of superposition is determined for each frame. As a result of the determination, if it is determined that there is superposition, the process proceeds to step S1105. On the other hand, if it is determined that there is no overlap, the process proceeds to step S1121.

  In step S <b> 1105, the cut start position, the cut width, and the height of the cut noise data 502 are determined from the scratch flaw noise data 501 stored in the memory 210. By changing these, the pattern (length, strength, number, etc.) of the scratches to be superimposed changes. Therefore, it may be determined using random number generation means so as to cut out from a different start position each time. When the cutout position is calculated, the process proceeds to step S1106.

  In step S1106, the cutout position determined in step S1105 is instructed to the cutout processing unit 206, and the process proceeds to step S1107.

  In step S1107, the superimposition target time is determined and stored in the temporary memory, and then the process proceeds to step S1108.

  In step S1108, addition (increment) of the superposition counter is performed, and the process proceeds to step S1109.

  In step S1109, a superposition continuation counter (second detection means) is added to measure the elapsed time since it is determined that there is no scratch scratch superposition, and the flow proceeds to step S1110.

  In step S1110, the position where the noise data cut out by the cut-out processing unit 206 is to be combined and pasted on the currently captured image is determined. The pasting position is a random pasting position depending on the timing of synthesis. For example, when a pasting position is acquired using random number generation means, scratches are superimposed at different positions on the screen every time the composition is performed, and it appears as random noise. When the pasting position is acquired, the process proceeds to step S1111.

  In step S1111, the current frame rate information of the image sensor 200 is acquired based on the drive timing signal instructed to the signal generator 203. If information is acquired, it will progress to step S1112.

  In step S1112, a determination reference value (predetermined value) for determining how far the current pasting position and the previous pasting position are located is calculated. Since the scratch scratch to be synthesized is updated in synchronization with the frame rate, the scratch scratch update cycle becomes longer as the frame rate decreases, that is, the captured image update cycle becomes longer, and the scratch scratch continues to be seen. If the distance between the previous pasting position and the current pasting position is large, the combined scratches appear to move. Therefore, the criterion for determining how far the current and previous pasting positions are is changed according to the frame rate information acquired in step S1111. For example, at a frame rate of 60 fps (first frame rate), when the current position is at a position farther than 10 pixels (first predetermined value) before and after the horizontal position with the previous position as the center, a far position that is larger than the predetermined value. It is determined that On the other hand, at 30 fps (second frame rate), the criterion is such that if the current position is farther than 20 pixels before and after the horizontal (second predetermined value), it is determined that the position is farther than the predetermined value. To change. When the determination criterion is determined, the process proceeds to step S1113.

  In step S1113, the distance of how much the current pasting position calculated in step S1110 has moved from the previous pasting position of the noise data is calculated. If it calculates, it will progress to step S1114.

  In step S1114, the distance between the pasting positions calculated in step S1113 is compared with the determination criterion calculated in step S1113. If it is determined that the distance is greater than the criterion (predetermined value), that is, the pasting position has changed significantly from the previous time to this time, the process proceeds to step S1116. On the other hand, if it is determined that the distance is smaller than the criterion, that is, the pasting position has changed slightly from the previous time to this time, the process proceeds to step S1115.

  In step S1115, the superposition continuation counter upper limit value (second predetermined time) defined in advance is changed to be larger. If it changes, it will progress to step S1117.

  In step S1116, the superposition continuation counter upper limit value specified in advance is changed to be smaller. If it changes, it will progress to step S1117.

  In step S1117, the superposition continuation counter upper limit value is compared with the value of the superposition continuation counter stored in the temporary memory. If the superposition continuation counter reaches the upper limit, the process proceeds to step S1122. On the other hand, if the upper limit has not been reached, the process proceeds to step S1118.

  In step S1118, the determined pasting position is stored in a temporary memory and used for the next control. If it preserve | saves, it will progress to step S1119.

  In step S 1119, the current paste position is instructed to the paste processing unit 208. When the instruction is given, the process proceeds to step S1120.

  In step S1120, the composition processing unit 205 is instructed to validate the superimposition of the scratch scratch instructed to be pasted and the captured image. When superimposition is enabled, control is terminated.

  In step S1121, the value of the superposition continuation counter stored in the temporary memory is cleared to the initial state and stored. When the storage ends, the process proceeds to step S1122.

  In step S <b> 1122, the composition processing unit 205 is instructed to invalidate the superimposition of the scratch scratch instructed to be pasted and the captured image. The invalidation method may be a method in which only the captured image is output without performing the synthesis process itself, or a method in which only the captured image is output by reducing the synthesis ratio of the scratches to the captured image, although the synthesis process is performed. Well, any realization means may be used. When superimposing is disabled, the control is terminated.

  In the second embodiment, an arbitrary position is randomly extracted every time in accordance with the update of the captured image from the noise data composed of a plurality of patterns of scratches stored in advance in the storage unit, and this is captured image And at any position. As a result, it is possible to add to the moving image scratch scratches that occur randomly and temporally and spatially.

  Furthermore, when the addition of scratches continues for a certain period or longer, the current pasting position of noise data is compared with the previous pasting position. At this time, it is possible to realize an operation close to the actual occurrence of scratches by restricting the addition of scratches to a position where the movement of the position is far away.

  In the first and second embodiments described above, the grain noise cut-out position change control and the scratch scratch addition invalidity control depend on the frame rate of the moving image, and the grain noise change condition and the scratch damage control. The invalid condition was changed. However, when the synthesis ratio of noise data is small, it is difficult for the noise component to be visually recognized in the synthesized image. Therefore, in the grain noise cut-out control, even if the current cut-out position is near the previous position, it is difficult to recognize the pattern as moving.

  Therefore, it goes without saying that if the setting of the composite ratio of the captured image and noise data in the synthesis can be changed by a user operation or automatically, the condition can be changed according to the synthesis ratio of the noise data. That is, the smaller the synthesis ratio of noise data to the captured image is, the larger the predetermined value can be.

  In the first and second embodiments described above, the case where grain noise and scratches are added in the moving image shooting operation of the digital video camera has been described. However, the present invention is not limited to moving image shooting, and can be used for continuous shooting in still image shooting and adding different patterns of grain noise during each shooting. Needless to say, it can be used to add the effect of film shooting in the reproduction of moving images or still images.

<Third Embodiment>
In the present embodiment, an image processing apparatus capable of giving an image an up-and-down shake of a video generated when a film is played on a projector as a film-like effect is shown. FIG. 14 and FIG. 15 explain the image cut-out from the frame memory and the process of concealing the noise (data without image information) due to the cut-out in order to realize the vertical shake in this embodiment.

  FIG. 14 is a diagram showing the structure of data in the frame memory in which the captured image captured by the image sensor is stored and the image displayed when clipping is performed at a position specified from the frame memory. Data in the frame memory is updated sequentially. Although data for other purposes is stored in the areas before and after the captured image, the captured image data can be regarded as noise data (data without image information). The stored captured image is output as a display image 1400 when it is cut out at a predetermined cutout start position and within a predetermined cutout range. By determining the cutout start position with a random number, it is possible to cut out from different positions, and it is possible to output the captured images in which the vertical shake has occurred like the display images 1401 and 1402. The cutout start position is determined with one pixel in the vertical direction of the image, that is, one line as a minimum unit.

  In addition, various random numbers, such as a Gaussian distribution, are conceivable as the random number for determining the cutout start position, but it does not depend on the type.

  In addition, by setting an upper limit on the amount of vertical shake, that is, a state where there is no shake as a reference position, an upper limit is set for the offset amount from the reference position to the cutout start position, so that a certain amount of shake does not occur.

  Also, the sum of the two upper and lower shake amounts determined at different periods is used as an offset amount, and the image reading start position is shifted from the upper left corner by the amount of the offset amount, and is set as the cutout start position. By doing so, it is possible to express the vertical blur configured as a combination of different types of blur such as the vertical blur generated in the film feeding operation and the vertical blur generated in the film winding operation.

  FIG. 15 is a diagram illustrating a method for concealing noise data generated when the effect of vertical shake is added. Noise data is displayed at the lower part of the screen in the captured images 1501 and 1505 in which arbitrary vertical shake occurs with respect to the captured image 1500 having no vertical shake. As a concealing method, for example, a method of adding a mask or a method of enlarging can be considered.

  In the method of adding a mask, a mask image 1503 that is greater than or equal to the maximum blur width (maximum offset amount) of the top and bottom blur is superimposed on the lower portion of the screen to hide the noise data. At this time, by superimposing a mask image of the same size on the upper part of the screen, a captured image 1502 having a so-called letterbox aspect ratio can be displayed.

  On the other hand, in the method of enlarging, by enlarging the area 1506 that does not include the maximum blur width of the upper and lower shakes so that the height of the area 1506 is equal to the screen height and maintaining the aspect ratio of the screen, A display image 1507 can be displayed.

  FIG. 13 is a block diagram of a digital video camera as an image processing apparatus according to the third embodiment. The image sensor 1300 photoelectrically converts the incident light that has been imaged. The photoelectrically converted signal is input to the camera signal processing unit 1301. The camera signal processing unit 1301 performs various kinds of image processing on the photoelectrically converted signal, converts it into a video signal, and outputs it. The output video signal is encoded by the encoder unit 1311 in a predetermined recording format and recorded on the recording medium 1312.

  Next, signal processing in the camera signal processing unit 1301 will be described. The signal input to the camera signal processing unit 1301 is first subjected to various signal processing in the image processing unit 1304 as in the first embodiment. The processed signal is stored in the first frame memory 1305 as a captured image.

  The cutout processing unit 1306 cuts out an arbitrary position from the first frame memory 1305 with a predetermined upper limit value of the offset amount in order to give the effect of vertical shake.

  Reference numeral 1308 denotes a second frame memory that stores a mask image for hiding noise data generated by the clipping process. The mask processing unit 1307 generates a mask image that is equal to or greater than the maximum width of the upper and lower blur, and stores the mask image in the second frame memory 1308.

  The synthesis processing unit 1309 reads the captured image 1504 stored in the first frame memory 1305 and the mask image 1304 generated by the mask processing unit 1307 and stored in the second frame memory 1308 at a predetermined timing. Performs composition and outputs a composite image.

  A signal generator (SG) 1303 is a signal generator that generates a signal for controlling the drive timing of the image sensor 1300. The generated signal is supplied to the image sensor 1300 and the system controller 1302.

  A system controller 1302 controls a camera block including the image sensor 1300 and the camera signal processing unit 1301. A signal accumulation period and readout timing are instructed to the image sensor 1300. For the image processing unit 1304, parameters necessary for image quality setting are set by various signal processing. For the synthesis processing unit 1309, the frame memory to be synthesized is designated and the synthesis timing is instructed. The cutout processing unit 1306 is instructed about the timing of reading from the first frame memory 1305 and the cutout position. The mask processing unit 1307 is instructed to specify the size and color of the mask to be superimposed, and to start and cancel the operation. The signal generator 1303 is instructed to drive timing for driving the image sensor 1300.

  Next, with reference to FIGS. 16 and 17, a flowchart of the cutout process and the mask process of the captured image in which the system controller 1302 in the present embodiment instructs the cutout processing unit 1306 and the mask processing unit 1307 and each unit operates will be described. This control is performed in a shooting mode in which image processing is performed as if vertical shake is added to the shot image. Alternatively, the moving images stored in the memory such as the recording medium 1312 are continuously read out and performed as post processing. This processing is started in conjunction with the vertical synchronization signal, and a series of processing is repeated by the synchronization signal. In the present embodiment, whether or not to add vertical shake can be automatically switched by a user manually or by scene determination. When adding top / bottom shake, process the mask → top / bottom shake in order to prevent the noise data from being seen. To stop adding top / bottom shake, reverse top / bottom stop → mask release. .

  In step S1601, the control of the vertical shake stop request is performed. If the operating state such as the shooting mode information is acquired and it is detected that the shooting mode has changed from the shooting mode that requires up / down shaking to the unnecessary shooting mode, a stop request is issued, and the process proceeds to step S1609. Otherwise, the process proceeds to step S1602.

  In step S1602, the occurrence request for vertical shake is controlled. When the operating state such as the shooting mode information is acquired and it is detected that the shooting mode has changed from the shooting mode that does not require vertical shaking, a generation request is issued, and the process proceeds to step S1603. Otherwise, the process proceeds to step S1612.

  In step S1603, the operation state of the vertical shake is set to “occurrence”. When the state is “occurrence”, it is possible to control vertical movement. When the setting is completed, the process proceeds to step S1604.

  In step S1604, the operating state of the mask to be added vertically is determined. If the result of determination is “addition”, the process proceeds to step S1605, and if it is “release” instead of “addition”, the process proceeds to step S1607.

In step S1605, the amount of vertical shake is determined. The amount of blur is a value determined randomly, for example, for the value obtained from the random number generator,
X = Xmax × (r / R)
It may be determined by the following formula. Here, X is a shake amount, Xmax is a maximum shake amount, r is a value obtained from the random number generation means, and R is a maximum value output from the random number generation means.

Also, as described above, a plurality of different types of blur can be combined. For example, a combination of a shake that changes in a short period but a small amount of blur as the first shake and a shake that changes in a long period but a large amount of shake as the second shake is considered. The time variation of the first shake is indicated by 1701 in FIG. 17, and similarly, the time change of the second shake is indicated by 1702 in FIG. At this time, the sum of the temporal changes of the first and second shake amounts, which is a combination of the two shakes, is shown by 1703 in FIG. At this time, the shake amount X is a value obtained from the random number generation means,
X = Xmax1 * (r1 / R1) + Xmax2 * (r2 / R2)
It may be determined by the following formula. Here, Xmax1 is the maximum shake amount in the first shake amount change, Xmax2 is the maximum shake amount in the second shake amount change, r1 and r2 are values obtained from the random number generator as the first and second shake amounts, and R1 , R2 is the maximum value output by the random number generating means.

  If the shake amount is determined in step S1605, the process proceeds to step S1606.

  In step S1606, the cutout start position of the captured image stored in the frame memory is calculated from the amount of blur determined in step S1605. When the calculation is finished, the control is finished.

  In step S1607, the mask processing unit 1307 is caused to add a mask image that conceals the noise data at the bottom of the screen that can be seen when vertical shake occurs. At this time, size information indicating the size of the mask image is instructed to the mask processing unit 1307. When the addition of the mask image ends, the process proceeds to step S1608.

  In step S1608, the operation state of the mask is set to “addition”, and the control ends.

  In step S1609, the operating state of the vertical shake is determined. If the state is “stop”, the process proceeds to step S1610. If the state is “occurrence”, the process proceeds to step S1612.

  In step S1610, the mask processing unit 1307 is instructed to cancel the addition of the mask image that hides the noise data at the bottom of the screen that is visible when the vertical shake occurs. The instructing information is superimposition invalid information that invalidates the superimposition of the mask image. When the instruction is finished, the process proceeds to step S1611.

  In step S1611, the operation state of the mask is set to “release”, and the control ends.

  In step S1612, the movement state of the vertical shake is set to “stop”. When the state is “stop”, the vertical shake control is disabled. When the setting is completed, the process proceeds to step S1613.

  In step S1613, a value with no up and down shake is set as the shake amount. When the shake amount is determined, the process proceeds to step S1614.

  In step S1614, the cutout start position of the captured image stored in the frame memory is calculated from the amount of blur determined in step S1613. At this time, the cutout start position is a reference position. When the calculation is finished, the control is finished.

  Looking at the above steps by situation, (1) there is no shake stop request when up / down shake addition starts, and there is a shake occurrence request, so the process proceeds to S1601, S1602, S1603, and S1604. In S1604, since the upper and lower mask states are in the initial state, the upper and lower mask states are released, the process proceeds to S1607. First, the upper and lower masks are added. In S1608, the upper and lower mask states are changed to the added state, and the first round is completed. In the next frame, the process proceeds to S1604 in the same manner, and the upper and lower mask states = added. Next, in the processes in S1605 and S1606, vertical shake is generated. Thereafter, (2) when addition of up and down movement is continued, the processing of S1601 to S1606 is repeated for each frame. Further, (3) when the addition of up and down shake is finished, there is a shake stop request, no shake occurrence request is made, and the process proceeds from S1601 to S1609. In the first round, since the up / down shake state = occurrence state, the process proceeds to S1612. The up / down shake is stopped, and the up / down shake is first stopped in S1613 and S1614. In the next frame, the process proceeds to S1601 and S1609. Since the up / down shake state = stopped state, the process proceeds to S1610, the release of the upper / lower mask is executed, the upper / lower mask state = release is completed, and the second round is completed. End addition of. (4) In the case where the state in which the vertical shake is not added is continued, since there is no shake stop request and no shake occurrence request, the process proceeds from S1601 to S1602, and then to S1612. Since S1613 and S1614 are steps for creating a cut-out position with no shake amount, that is, a state in which there is no vertical shake, by continuing this loop, a state without vertical shake is continued.

  As described above, in the third embodiment, an image obtained by clipping an image at a randomly determined clipping start position is displayed, and this occurs when the film is played on a projector as a film-like effect. This makes it possible to give the image the top and bottom blur of the video to be played.

<Fourth Embodiment>
In the present embodiment, an image processing apparatus capable of giving an image an unevenness in brightness of a video (hereinafter referred to as blinking) that occurs when a film is played on a projector as a film-like effect is shown.

  FIG. 19 illustrates luminance signal correction processing for realizing blinking in the present embodiment. In the present embodiment, a method of performing correction to change the input / output characteristics of the luminance signal is used in order to finally realize blinking of an image to be displayed and recorded. In the correction process, the luminance component of the output image is sequentially changed with different input / output characteristics with respect to the input captured image. In the present embodiment, the minimum unit of change in the luminance signal is the minimum resolution of the signal of the output image, and the minimum unit of time is the update period of the captured image.

  When correction is performed in the order of the characteristics indicated by the solid lines of the characteristics 1901 to the characteristics 1902, 1903, and 1904, the captured images become images 1906, 1907, and 1908 with respect to the image 1905 corresponding to the characteristics 1901. At this time, the images are bright in the order of image 1908> image 1905> image 1907> image 1906.

  In order to generate blinking at random, for example, it is conceivable to prepare a plurality of parameter data constituting the input / output characteristics and determine which parameter data is to be used from among the parameter data. There are various random numbers for determining the parameter data, including a Gaussian distribution, but it does not depend on the type.

  FIG. 18A is a block diagram of a digital video camera as an image processing apparatus according to the fourth embodiment. The image sensor 1800 photoelectrically converts the incident light that has been imaged. The photoelectrically converted signal is input to the camera signal processing unit 1801. The camera signal processing unit 1801 performs various kinds of image processing on the photoelectrically converted signal, converts it into a video signal, and outputs it. The output video signal is encoded by the encoder unit 1808 in a predetermined recording format and recorded on the recording medium 1809.

  Next, signal processing in the camera signal processing unit 1801 will be described. The signal input to the camera signal processing unit 1801 is first subjected to various signal processing by the image processing unit 1804 for each of the luminance component, color component, and black component. The processed signal is stored in the frame memory 1805 as a captured image.

  A characteristic data storage unit 1807 is a memory that stores a plurality of parameter data for determining the input / output characteristics of the luminance of the video signal. For example, when the characteristic is expressed by a linear function such as 1901 to 1904, the parameter data includes a slope of a straight line and an intercept. Also, as in the characteristic 1904, when the input is large, the point at which the output is clipped and the value to be clipped are also parameter data. The determined parameter data is sent to the correction processing unit 1806.

  The correction processing unit 1806 determines luminance input / output characteristics according to the parameter data sent from the characteristic data storage unit 1807, corrects the captured image stored in the frame memory 1805 according to the characteristics, and corrects the corrected image. Is output. In addition, since the correction is performed after the image signal is processed by the image processing unit 1804 after the luminance signal (Y signal) and the color difference signals (U and V signals), even if the output characteristics change due to the correction, the result is the image processing. The luminance component processing performed by the unit 1804 is not affected.

  A signal generator (SG) 1803 is a signal generator that generates a signal for controlling the drive timing of the image sensor 1800. The generated signal is supplied to the image sensor 1800 and the system controller 1802.

  A system controller 1802 controls a camera block configured by an image sensor 1800 and a camera signal processing unit 1801. A signal accumulation period and readout timing are instructed to the image sensor 1800. For the image processing unit 1804, parameters necessary for image quality setting are set by various signal processing. For the characteristic data storage unit 1807, input / output characteristic parameter data is designated. The correction processing unit 1806 is instructed to read data from the frame memory 1805 and instruct the correction processing to be valid / invalid. The signal generator 1803 is instructed to drive timing for driving the image sensor 1800.

  Next, with reference to FIG. 20, a flowchart of correction processing and characteristic data determination processing in which each unit operates when the system controller 1802 in this embodiment instructs the correction processing unit 1806 and the characteristic data storage unit 1807 will be described. This control is performed in the shooting mode in which image processing is performed as if blinking is added to the shot image. Alternatively, the moving images stored in the memory such as the recording medium 1809 are continuously read and performed as post processing. This processing is started in conjunction with the vertical synchronization signal, and a series of processing is repeated by the synchronization signal.

  In step S2001, first, the operation state of blinking control is acquired from the current shooting mode information and the like, and the result is determined. If it is determined that “occurrence” of blinking is necessary from the acquired operation state, the process proceeds to step S2002, and if it is determined that “stop” of blinking is necessary from the operation state, the process proceeds to step S2004.

  In step S2002, parameter data for changing the input / output characteristics stored in the characteristic data storage unit 1807 is determined in order to cause blinking. At this time, in order to change the blinking amount, one of the plurality of parameter data is selected. For example, the parameter data is determined based on a value randomly obtained from the random number generation means in order to randomly generate the blinking amount. When the parameter data is determined, the process proceeds to step S2003.

  In step S2003, the correction processing unit 1806 performs correction processing using the parameter data determined in step S2002, and the processing ends.

  In step S2004, reference parameter data is selected from the input / output characteristics stored in the characteristic data storage unit 1807 to stop blinking. Thus, the same characteristics are always obtained when blinking is stopped. When the selection is completed, the process proceeds to step S2003.

  As described above, in the fourth embodiment, correction for changing the input / output characteristics of the luminance signal is performed on the image in addition to the normal image processing. As a result, it is possible to give the image unevenness in the brightness of the video that occurs when the film is played on a projector as a film-like effect.

  FIG. 14B is a block diagram showing an embodiment in which correction processing that gives a blinking effect is performed collectively in gamma processing in the image processing unit 1804 that is also performed during normal shooting. The white balance processing unit 1810 performs white balance processing and outputs red (R), green (G), and blue (B) signals at the output stage. Upon receiving this signal, the correction processing unit 1806 receives the luminance signal. The process is divided into color signals. Specifically, RGB signals for luminance signals and RGB signals for color signals are prepared, and gamma correction suitable for each is applied. Thereafter, a luminance (Y) signal is generated from the RGB signal for luminance signal, a color difference (Cb, Cr) signal is generated from the RGB signal for color signal, and is output to the color balance correction unit 1811. In these processes, the gamma curve applied to the RGB signal for the luminance signal is converted into a curve taking into account the input / output characteristics in the present embodiment. Thereby, the blinking effect can be realized without requiring a separate processing block or memory for blinking.

<Fifth Embodiment>
In this embodiment, as image processing that gives a video effect that occurs in an image when the film is played on a projector, the image that is given to the image by overlapping the scratches, blinking, and vertical blur described in the previous embodiment. 1 shows an image processing apparatus capable of executing processing. As a method of giving each effect of scratching, blinking, and vertical shaking, the method in the above-described embodiment is used, and detailed description thereof is omitted here.

  FIG. 21 is a block diagram of a digital video camera as an image processing apparatus according to the fifth embodiment. The image sensor 2100 photoelectrically converts the incident light that has been imaged. The photoelectrically converted signal is input to the camera signal processing unit 2101. The camera signal processing unit 2101 performs various kinds of image processing on the photoelectrically converted signal, converts it into a video signal, and outputs it. The output video signal is encoded in a predetermined recording format by the encoder unit 2113 and recorded on the recording medium 2114.

  Next, signal processing in the camera signal processing unit 2101 will be described. The signal input to the camera signal processing unit 2101 is first subjected to various signal processing in the image processing unit 2104 as in the first embodiment, and is output as a luminance signal and a color difference signal. The processed signal is stored in the first frame memory 2105 as a captured image.

  Reference numerals 2106, 2107, and 2108 denote second, third, and fourth frame memories that temporarily store the results of various processes that give a film-like effect.

  Next, a processing unit that gives a film-like effect will be described. Reference numeral 2109 denotes a cut-out processing unit that cuts out an image at a predetermined cut-out position in order to generate up and down blur, and reference numeral 2110 denotes a correction processing unit that corrects the input / output characteristics of the luminance of the image to cause blinking. . Reference numeral 2111 denotes a synthesis processing unit that synthesizes a scratch scratch noise image with an image in order to add a scratch, and reference numeral 2112 denotes a mask processing unit that adds a mask image for concealing noise generated during the clipping process.

  A signal generator (SG) 2103 is a signal generator that generates a signal for controlling the drive timing of the image sensor 2100. The generated signal is supplied to the image sensor 2100 and the system controller 2102.

  A system controller 2102 controls a camera block composed of an image sensor 2100 and a camera signal processing unit 2101. A signal accumulation period and readout timing are instructed to the image sensor 2100. For the image processing unit 2104, parameters necessary for image quality setting are set by various signal processing. For the cutout processing unit 2109, instructions for enabling / disabling cutout control, instructions for a cutout position from the first frame memory 2105, reading from the first frame memory 2105, and writing to the second frame memory 2106 The instruction of the timing to perform is performed. Instructions for enabling / disabling correction processing, specifying parameter data of input / output characteristics, reading from the second frame memory 2106, and writing to the third frame memory 2107 to the correction processing unit 2110 I do. For the synthesis processing unit 2111, instructions for enabling / disabling synthesis control, designating a noise image (not shown) to be synthesized, reading from the third frame memory 2107, and writing to the fourth frame memory 2108 are performed. Instruct the timing to do. For the mask processing unit 2112, a mask size and color instruction to be superimposed, a timing to read from the frame memory 2108, and an operation start and release instruction are given as mask image control. The signal generator 2103 is instructed to drive timing for driving the image sensor 2100.

  Next, FIG. 22 shows a flowchart of the control performed by the system controller 2202 in the present embodiment on the cut-out processing unit 2109, the correction processing unit 2110, the synthesis processing unit 2111, and the mask processing unit 2112 and the operations of the respective units. This control is performed in the shooting mode in which image processing is performed as if a plurality of film-like effects are added to the shot image. Alternatively, the moving images stored in the memory such as the recording medium 2114 are continuously read out and performed as post processing. This processing is started in conjunction with the vertical synchronization signal, and a series of processing is repeated by the synchronization signal.

  In step S2201, information on the shooting mode and various settings is acquired, and it is determined whether vertical shake is valid or invalid. If it is determined that the vertical shake is valid, the cutout processing unit 2109 is instructed to execute cutout processing, and the process advances to step S2202. On the other hand, if it is set to invalid, the cut processing unit 2109 is instructed not to execute cut control, and the process advances to step S2203.

  In step S2202, clipping processing is performed on the image data read from the first frame memory 2105 as shown in the third embodiment. Mask processing is performed similarly to the processing in S1605 and S1606 in the flow of FIG. The cut out image data is written into the second frame memory 2106. Then, the process proceeds to step S2203.

  In step S2203, information on the shooting mode and various settings is acquired, and whether flashing is enabled or disabled is determined. If it is determined that blinking is valid, the correction processing unit 2110 is instructed to execute correction processing, and the process advances to step S2204. On the other hand, if it is determined as invalid, the correction processing unit 21010 is instructed not to perform correction processing, and the process advances to step S2205.

  In step S2204, blinking is controlled by correcting the input / output characteristics of the luminance as shown in the fourth embodiment for the image data read from the second frame memory 2106. In accordance with the various settings, the correction processing unit 2110 advances one round process according to an instruction from the system controller 2202 along the flow of FIG. The corrected image data is written into the third frame memory 2107, and the process proceeds to step S2205.

  In step S2205, information on the shooting mode and various settings is acquired, and the validity / invalidity of adding scratches is determined. If it is determined that the addition of scratches is valid, the compositing processing unit 2111 issues an instruction to perform synthesis control of the scratched noise image, and the process advances to step S2206. On the other hand, if it is determined to be invalid, the compositing processing unit 2111 is instructed not to perform the control for synthesizing the scratched noise image, and the process advances to step S2206.

  In step S <b> 2206, the image data read from the third frame memory 2107 is subjected to a process of combining scratch noise image data as described in the second embodiment. In accordance with the flow of FIG. 11, the synthesis processing unit 2111 advances one round process according to an instruction from the system controller 2202 according to various settings. The cut-out position and synthesis position of the noise image are determined. The synthesized image data is written into the fourth frame memory 2108, and the process proceeds to step S2207.

  In step S2207, information on the shooting mode and various settings is acquired, and it is determined whether the upper and lower masks are added or not. If it is determined that the addition of the upper and lower masks is valid, the mask processing unit 2112 is instructed to execute the mask image synthesis control, and the process advances to step S2208. On the other hand, if it is determined to be invalid, the mask processing unit 2112 is instructed not to perform mask image synthesis control, and the process advances to step S2208.

  In step S2208, processing for adding a mask image is performed on the image data read from the fourth frame memory 2108, as described in the fourth embodiment. Mask processing is performed similarly to the processing in S1607 and S1608 in the flow of FIG. The masked image data is output to the encoder unit 2113 and the like as image data after various signal processes are performed by the camera signal processing unit 2101, and the first round of processing is completed.

  As described above, in the fifth embodiment, scratch damage, blinking, and up / down blur are superimposed on an image of input image data and applied to the image. As a result, it is possible to more faithfully give a video effect that occurs in an image when the film is played on a projector.

  FIG. 21B shows an image processing apparatus according to an embodiment in which the effects given to the image in the present embodiment are collectively given in parallel with various processes in the normal shooting mode. . In this form, the cutout processing unit 2109 normally buffers the image read from the sensor once in the frame memory, performs angle-of-view adjustment such as centering and unnecessary pixel cut, and is a block that performs these cutout processes. is there. It is also used as an electronic image stabilization function for correcting blurring of the entire image due to camera shake. In the present embodiment, the cut-out process for vertical shake is also executed here. Further, as in FIG. 18B, the correction processing unit 2110 that performs gamma correction to the luminance signal that is performed in the image processing unit 2104 even during normal photographing is corrected for the luminance input / output characteristics for blinking. Gamma correction is executed by the gamma curve based on the above. As a result, the number of newly provided processing blocks is reduced, and the processing speed can be increased.

<Sixth Embodiment>
In the present embodiment, in order to superimpose a video effect (film-like effect) generated on an image when the film is played on a projector on a digital image, grain noise and scratches are added in an overlapping manner. An image processing apparatus capable of Details on the generation of grain noise and scratches are described in detail in the first and second embodiments, and are omitted here.

  FIG. 24 is a diagram illustrating a process from recording (hereinafter referred to as shooting) to playback (hereinafter referred to as screening) in moving image shooting using a film as a recording medium. FIG. 25 shows events that affect the screening results ((A) and (B) in the process of (1)) and effects by the events (also (A-1) and (A -2), (B)), and a table on which means for realizing the effect by pseudo image processing are placed.

  The process of moving image shooting is divided into four steps: (1) shooting, (2) development / editing, (3) storage, and (4) screening. Grain noise is generated by (2) development / editing process, for example, by a development technique known as “silver strain”. Scratch scratches (4) occur when the running film is scratched during the screening process. That is, grain noise and scratches are noises with different characteristics that are generated due to other causes in the process in which an image taken with a film is shown. In the first embodiment, the second embodiment, and the present embodiment, these two noises are realized by synthesizing a noise image simulating generated noise with a captured image.

  In addition, in the process of (1) shooting, (A-1) the peripheral light amount drops and (A-2) distortion occurs due to the characteristics of the (A) optical system used for shooting. Further, (3) during the storage process, (A) fading due to aging of the film, (B) adhesion of dust, noise due to occurrence of stains, and the like are generated. Similarly, for these effects, (1) (A-2) deformation of the display image in the image processing apparatus if it is distorted, and (B) random grain noise in (2) or (3) if it is dust. By pasting, the image can be given in a pseudo manner. Further, if the color is faded, (3) (A) color balance and color gain correction.

  When adding a plurality of effects as film-like effects, a more preferable result can be obtained by giving the effects in the order corresponding to the film photographing process. When both grain noise and scratches are added, the development / editing process that causes grain noise is in the previous stage of the screening process that causes scratches. For this reason, in the present embodiment, a process of adding a scratch to the image data obtained by combining the grain noise with the input image data is performed. This makes it possible to give a more faithful film-like effect.

  FIG. 23 is a block diagram of a digital video camera as an image processing apparatus according to the sixth embodiment. The image sensor 2300 photoelectrically converts the incident light that has been imaged. The photoelectrically converted signal is input to the camera signal processing unit 2301. The camera signal processing unit 2301 performs various kinds of image processing on the photoelectrically converted signal, converts it into a video signal, and outputs it. The output video signal is encoded by the encoder unit 2314 in a predetermined recording format and recorded on the recording medium 2315.

  Next, signal processing in the camera signal processing unit 2301 will be described. The signal input to the signal processing unit 2301 is first subjected to various types of signal processing in the image processing unit 2304 as in the first embodiment. The processed signal is stored in the first frame memory 2305 as a captured image.

  Reference numeral 2308 denotes a scratch flaw generator (selection unit) that generates and cuts out a scratch flaw noise image according to an instruction from the system controller 2302 (second specifying unit). The noise image (second noise data) stored in the second frame memory 2306 (second storage means) is processed as shown in the second embodiment and is synthesized by the synthesis processing unit 2310.

  Reference numeral 2309 denotes a grain noise generation unit (cutout unit) that generates and cuts out a noise image of grain noise in accordance with an instruction from the system controller 2302 (first designation unit). The noise image (first noise data) stored in the third frame memory 2307 (first storage means) is processed as shown in the first embodiment and synthesized by the synthesis processing unit 2310.

  A composition processing unit 2310 is a composition processing unit capable of sequentially compositing the captured image stored in the first frame memory 2305 with two different images. Reference numeral 2311 denotes a first combining unit that combines the first input images, and combines the image data stored in the first frame memory 2305 and the noise image data of grain noise. Reference numeral 2312 denotes a second synthesis unit that synthesizes the second input image, and synthesizes the image data of the output result of the first synthesis unit and the noise image data of the scratch. The switching control unit 2313 switches whether to input image data to be combined to the first and second combining units. Input switching can be set independently in each of the first and second combining units. Further, when not input to each of the first and second combining units, the image data stored in the first frame memory 2305 or the image data output from the first combining unit is output as it is. .

  A signal generator (SG) 2303 is a signal generator that generates a signal for controlling the drive timing of the image sensor 2300. The generated signal is supplied to the image sensor 2300 and the system controller 2302.

  A system controller 2302 controls a camera block composed of an image sensor 2300 and a camera signal processing unit 2301. A signal accumulation period and readout timing are instructed to the image sensor 2300. In the image processing unit 2304, parameters necessary for image quality setting are set by various signal processing. For the synthesis processing unit 2310, an instruction for reading the captured image of the frame memory 2305 to be synthesized, an instruction for a synthesis ratio of each of the first and second synthesis units, and each of the first and second synthesis units are synthesized. Instruct whether or not to perform. In order to generate a scratch, the scratch scratch generation unit 2308 is instructed to read and write noise data to the memory 2306, instruct the cut position and size, instruct the size at the time of resizing, and at the time of synthesis. Specify the pasting position. For the grain noise generation unit 2309, in order to generate grain noise, instructions for reading / writing the noise data to / from the memory 2307, instructions for the cutout position and size, instructions for the size at the time of resizing, and information at the time of synthesis Specify the pasting position. The signal generator 2303 is instructed to drive timing for driving the image sensor 200.

  Next, FIG. 26 shows a composite image switching control instructed by the system controller 2302 in the present embodiment to the composition processing unit 2310, and a flowchart of processing performed in each unit under the control. This control is performed in a shooting mode in which a plurality of film-like noise effects are added to a shot image. Alternatively, the moving images stored in the memory such as the recording medium 2315 are continuously read out and performed as post processing. This processing is started in conjunction with the vertical synchronization signal, and a series of processing is repeated by the synchronization signal.

  When combining grain noise and scratches in a multiplexed manner, as described above, combining grain noise and scratches in this order provides an effect close to that when the film is actually screened. Therefore, the noise image output from the grain noise generation unit 2309 is input as a synthesis image of the first synthesis unit, and the noise image output from the scratch scratch generation unit 2308 is input as a synthesis image of the second synthesis unit. input.

  In step S2601, the type of noise effect added to the captured image stored in the frame memory 2305 is acquired, and the process advances to step S2602.

  In step S2602, it is determined whether grain noise is included in the type of noise effect acquired in step S2601. If it is included, the process proceeds to step S2603, and if it is not included, the process proceeds to step S2604.

  In step S2603, the synthesis of grain noise is validated, and the grain noise synthesis process is performed as shown in the first embodiment. In accordance with the various settings, the one-round processing is advanced by the grain noise generation unit 2309 and the synthesis processing unit 2310 according to instructions from the system controller 230 according to various settings. In the present embodiment, the first combining unit 2311 combines the input image data from the first frame memory 2305 and the grain noise data. The combined output result is output to the second combining unit 2312, and the process proceeds to step S2605.

  In step S2604, a setting for invalidating the synthesis of grain noise is determined, and the process proceeds to step S2605.

  In step S2605, it is determined whether the noise effect type acquired in step S2601 includes a scratch. If it is included, the process proceeds to step S2606, and if it is not included, the process proceeds to step S2607.

  In step S2606, scratch scratch synthesis is enabled, and scratch scratch noise synthesis processing is performed as shown in the second embodiment. In accordance with the various settings, one cycle processing is advanced by the scratch flaw generating unit 2308 and the synthesis processing unit 2310 according to instructions from the system controller 230 according to various settings. In this embodiment, scratch scratch noise data is generated by the scratch scratch generation unit 2308, and image data of the output result of the first combining unit 2311, scratch scratch noise data, and the second combining unit 2312. Is synthesized. The synthesized image data of the output result is output as the output of the camera signal processing unit 2301, and the control is terminated.

  In step S2607, the setting for invalidating the synthesis of scratches is determined, and the control ends.

  As described above, in the sixth embodiment, two noises, namely, grain noise and scratch damage, which are different in temporal and spatial randomness, are separately generated as noise data for the input image data image. Then, the image data is superimposed and synthesized. As a result, it is possible to more faithfully give a video effect that occurs in an image when the film is played on a projector. Furthermore, the order of synthesizing each noise with the image data is set to the order corresponding to the film photographing process, so that a more faithful noise effect can be obtained.

  In the present embodiment, the noise image data of grain noise and the noise image data of scratches are combined in order with the input image data. However, the present invention is not limited to this, and it is possible to combine the image data of scratch noise with the noise image data of grain noise and the input image data. At this time, when synthesizing the noise image data of the grain noise and the noise image data of the scratch damage, the scratch damage is synthesized (pasted) at 0: 1 by making the grain noise transparent in advance. Then, when the input image data and the synthesized noise image data are synthesized at a ratio of 0: 1, the grain noise is synthesized at a synthesis ratio corresponding to the transparency, and the scratches are pasted from above. An image is output, and a more faithful image is completed as in this embodiment.

(Other embodiments)
In each of the above-described embodiments, as an example of the image processing apparatus, an embodiment in which a film-like effect is given to a captured image of a video camera is shown. However, the present invention is not limited to this, and the present invention may be in any form as long as it is an image processing apparatus that can perform image processing on input image data. For example, various devices such as a digital still camera, a PC, and a portable terminal are conceivable.

  FIG. 27 shows an example of an embodiment in which processing is performed on a recorded image among image processing apparatuses to which the present invention can be applied. The signal processing unit 2701 corresponds to the camera signal processing unit in each of the above embodiments, and includes a frame memory and the like as necessary. The image data recorded in the recording unit 2703 is subjected to decoding or expansion processing as necessary by the recording control unit 2702, and processing for adding a film-like effect is performed by the signal processing unit 2701. The processed image data is sent again to the recording control unit 2702, encoded and compressed as necessary, and recorded in the recording unit 2703. Further, it is sent to the display control unit 2704 and displayed on the display unit 2705. Note that the configuration other than the signal processing unit 2701 is not necessarily in the image processing apparatus, and the recording unit 2703 may be a recording medium of an external device or an online storage on a WEB. The display unit 2705 may also be a display device such as an external monitor.

  The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software in which a procedure for realizing the functions of the above-described embodiments is described is recorded is supplied to the system or apparatus. The computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

  Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can also be used.

  Further, by making the program code read by the computer executable, the functions of the above-described embodiments are realized. Furthermore, when the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, the functions of the above-described embodiments are realized by the processing. Is also included.

  Furthermore, the following cases are also included. First, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

100, 200 Image sensor 101, 201 Camera signal processing unit 102, 202 System controller 103, 203 Signal generator 104, 204 Image processing unit 105, 201 Compositing processing unit 106, 206 Cutout processing unit 107, 207 Enlargement processing unit 108 First Frame memory 109, 210 memory 110 second frame memory 111, 211 encoder unit 112, 212 recording medium 208 pasting processing unit 209 frame memory

Claims (15)

Acquisition means for acquiring images of a plurality of frames;
For each of the images of the plurality of frames, clipping means for clipping at a position determined at random;
Mask means for superimposing a mask image on the cut-out image;
An image processing apparatus comprising: The cutting means cuts out with a predetermined size starting from a randomly determined position,
The image processing apparatus according to claim 1, wherein the mask unit superimposes a mask image having a predetermined size on the clipped image.   The cutout unit determines an offset amount based on two vertical shakes having different periods, and shifts an image reading start position by the offset amount to set the position as the position. 2. The image processing apparatus according to 2.   4. The image processing according to claim 1, further comprising: a luminance correction unit configured to perform correction by changing luminance input / output characteristics at random on the images of the plurality of frames. 5. apparatus.   The luminance correction means for performing correction by selecting a parameter at random from the parameters constituting the input / output characteristics of a plurality of luminances stored in advance for the images of the plurality of frames. 4. The image processing device according to any one of 3. The acquisition means captures and acquires the images of the plurality of frames,
6. The image according to claim 4, wherein the luminance correction unit performs correction for changing the input / output characteristics of luminance randomly in time with the gamma processing on the image of the plurality of frames. Processing equipment. Luminance correction means for performing correction that changes the input / output characteristics of the luminance at random with respect to the images of the plurality of frames,
The image processing apparatus according to claim 1, wherein the mask image is superimposed by the mask unit on the image corrected by the luminance correction unit. Having a synthesis means for superimposing noise data on an image;
The noise data is synthesized by the synthesizing unit with respect to the image whose luminance has been corrected by the luminance correcting unit, and then the clipping by the clipping unit and the mask image by the masking unit are superimposed. The image processing apparatus according to any one of 4 to 6.   An imaging unit that captures an image of a subject and sequentially outputs the images of the plurality of frames to the acquisition unit; and a recording unit that records the images of the plurality of frames output from the mask unit on a recording medium as a moving image. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Acquisition means for acquiring images of a plurality of frames;
Distortion processing means for performing a process of distorting the image of the plurality of frames;
Noise processing means for performing a process of adding noise to a plurality of frames of images processed by the distortion processing means;
A blur addition processing means for performing a process of adding blur to an image of a plurality of frames processed and output by the noise processing means;
Recording means for recording a plurality of frames of images processed by the blur addition processing means on a recording medium as moving images;
An image processing apparatus comprising: An acquisition step of acquiring a multi-frame image;
Cutting each of the images of the plurality of frames at a randomly determined position;
And a mask step for superimposing a mask image having a set size on the cut-out image. An acquisition step of acquiring a multi-frame image;
A distortion processing step of performing a process of distorting the image on the plurality of frames of images; a noise processing step of applying a process of adding noise to the images of the plurality of frames processed in the distortion processing step;
A blur addition processing step for performing a process of adding blur to a plurality of images to be processed and output in the noise processing step;
An image processing method comprising: a recording step of recording a plurality of frames of images processed in the blur addition processing step as a moving image on a recording medium. The noise processing step is a process of adding grain noise corresponding to the entire image.
The image processing method according to claim 12, wherein the blur addition processing step performs a process of cutting out at a position determined at random with respect to the image and in a cutout range. A computer-executable program in which the procedure of the image processing method according to any one of claims 11 to 13 is described. A computer-readable storage medium storing a program in which the procedure of the image processing method according to claim 11 is described.

Images