CA2244045A1 - Modifying image data - Google Patents

Abstract

An image data modifying system has devices for image storage, display, processing and manual control. The storage device is arranged to store many modifying textures in the form of stamps having fewer pixels than a full image frame. The processing device constructs a new texture stamp by scaling a first stamping texture, expanding a second stamping texture without scaling and combining the scaled texture with the expanded texture. In this way, the effect of scaling artefacts is significantly reduced.

Description

CA 0224404~ 1998-07-28 Modifying Image Data Field of the Invention The present invention relates to image data modifying apparatus, 5 comprising image storage means, image display means, processing means and a manually controllable input means. In addition, the present invention relates to a method of modifying image data in which pixel values in an image are modified in response to manual operation of an input device.

10 Background of the Invention The manual modification of image data within a data processing environment or video editing environment etc. is commonly referred to as "painting". While no physical paint is actually involved in this process, the operation performed by an artist, using a mouse, stylus or similar manually operable device, is similar to the action of applying a brush to a canvas and measures are often adopted within the environment to copy the response of brushes and similar graphical tools.
Painting processes are often implemented by performing a sequence of stamping operations in which a region of pixels Iying underneath the 20 notional footprint of a stamp are modified in response to values stored within a stamp patch, in addition to other control values which may be derived from stylus pressure etc. It is also known to store a plurality of stamp patches, each representing a different increment of sub-pixel brush movement, such that the effect of a natural brush stroke with a definition in excess of pixel 25 resolution may be accurately simulated.
As the definition of images increases, such as when modifying high definition video or when modifying images derived from cinematographic film, the number of pixels covered within a brush stamp will also increase proportionally and, in systems where a particular profile is recorded for each CA 0224404~ 1998-07-28 possible size, a significant number of brush profiles will need to be made available; although it is unlikely that all of these would actually be required for a particular job.
In order to overcome this problem and to avoid the need for storing a very large number of brush profiles, it is known to store an example of the brush profile representing a particular shape and to produce the range of sizes by scaling the brush profile. Furthermore, it is also known to manipulate brush profiles in the form of textures which may then be rendered into an existing frame buffer image at sub-pixel resolution. Thus, in this way, it is possible to store a single brush profile which may be used to produce high definition sub-pixel stamping in addition to producing a plurality of brush sizes, thereby drastically reducing the number of brush profiles which need to be stored. Thus, in a typical environment, it becomes only necessary to store a new brush texture if a differently shaped brush profile is required. Thus, brush profiles may be provided for simulating the effects of air brushes, stipple brushes, chalk, paint brushes and pencils etc. and the same data may be scaled and positioned at sub-pixel resolution so as to provide smooth anti-aliased stamping.
A problem with this approach is that as a brush stamp is scaled, many pixel values within the brush stamp will be given similar intensity values. As abrush of this type is used repeatedly, unwanted artefacts between brush values may become visible and possibly displayed in the form of a series of rings. Furthermore, the availability of processing resource for mitigating the effect of these rings is limited, given that the pixel processing is required inreal time, in response to manual indications made by an operator.

Summary of the Invention According to a first aspect of the present invention, there is provided an image data modifying apparatus comprising image storage means, image CA 0224404~ l998-07-28 display means, processing means and a manually controllable input means, wherein said storage means is arranged to store a plurality of modifying textures in the form of stamps having fewer pixels than a full image frame;
and said processing means is configured to construct new texture stamps by scaling a first stamping texture, expanding a second stamping texture without scaling and combining said scaled and said expanded textures.
In a preferred embodiment, the first stamping texture represents a brush profile and said storage means may be configured to store a plurality of brush profiles. Preferably, the second stamping texture represents random 10 noise and said random noise may have a relatively low amplitude.
According to a second aspect of the present invention, there is provided a method of modifying image data, in which pixel values in an image are modified in response to manual operation of an input device, comprising steps of storing a first stamping texture, storing a second stamping texture and producing a new stamping texture by scaling said first stamping texture while expanding said second stamping texture without scaling.
Preferably, a new texture is rendered into an image at sub-pixel definition.

Brief Description of the Drawings The invention will now be described by way of example only, with reference to the accompanying figures, in which:
Figure 1 illustrates a video artist performing a painting operation upon a video frame, including a manually operable device and a visual display unit;
Figure 2 details a main screen shown on the visual display unit, including buttons for selecting modes of operation, including a set-up mode;
Figure 3 details an image displayed on the visual display unit in response to selecting the set-up mode identified in Figure 2;

CA 0224404~ l998-07-28 Figure 4 illustrates painting stamps being applied to the displayed canvas shown in Figure 2;
Figure 5 illustrates a pixel texture map making up a relatively small stamp of the type shown in Figure 4;
Figure 6 shows how the relatively large stamp shown in Figure 4 is generated from the smaller texture map illustrated in Figure 5;
Figure 7 illustrates a noise texture map formed by tiling smaller texture maps; and Figure 8 illustrates how the scaled bush profile and the tiled random texture are combined in a painting operation performed by the apparatus illustrated in Figure 1.

Detailed Description of the Preferred Embodiment A facility for performing "painting" operations upon digital image data is illustrated in Figure 1. A processing system 101 is configured to store digital representations of image frames such that individual frame data may be displayed to an operator by means of a visual display unit 102. In response to this displayed image, an operator provides input data to the processing system 101 via a manually operable device, such as a stylus 103 arranged to 20. co-operate with a touch tablet 104. In addition, the operator is also provided with a conventional keyboard 105.
A typical image displayed on the VDU 102 is detailed in Figure 2. The display includes a canvas area 201 in which a visual image is created or an existing image is modified. Movement of stylus 103 is detected by system 101 which in turn displays a cursor 202 arranged to map movements of the stylus 103 over the touch tablet 104. Cursor 202 may be moved into the canvas area 201 while performing "painting" operations. In addition, the cursor 202 may also be placed over soft buttons which are activated by placing the stylus 103 into pressure. In this way, particular types of operation CA 0224404~ 1998-07-28 may be effected by selecting soft buttons 203, 204, 205, 206 and 207. Thus conventional painting operations may be selected by these buttons, in which a new colour is applied to the image or, alternatively, an operation may be selected in which a control plane, or matte, is modified so as to change the 5 way in which two image frames are blended.
The image screen displayed on monitor 101 also includes a palette area 208 allowing particular colours to be selected, along with buttons 209, 210, 211 and 212 allowing particular brush profiles to be selected. Thus, button 209 may select a sharp edged pencil, button 210 may select a softer 10 edged paintbrush, button 211 may select a stippled or textured brush and button 210 may select a very smooth edged airbrush.
Operational parameters of the system may be modified in response to activating system buttons, including buttons 213, 214 and 215. In this embodiment, the selection of button 215 allows the nature of the painting 15 operation to be modified and results in the screen shown in Figure 2 being replaced by a screen of the type shown in Figure 3.
The screen shown in Figure 3 again includes a plurality of buttons 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312 and 313. During operation of the stylus 103 it is possible to vary the pressure applied to the 20 stylus, modify the position of the stylus in the x direction and modify the position of the stylus in the y direction. These variables are expressed horizontally allowing relevant selections to be made from the matrix of buttons 301 to 313. These input variables may in turn adjust the intensity of the stamping, the size of the stamping, the rate at which stamping occurs and 25 the texture of the stamping.
In this way, when button 301 is activated, resulting in the way in which the button is represented being changed, the intensity of the brush stamp will vary in response to variations in applied pressure. These soft buttons are configured as toggles such that selecting button 301 again will result in the CA 0224404~ l998-07-28 button being switched off such that intensity will not vary with pressure.
Similarly, activation of button 302 will result in the intensity varying with respect to movement in the x direction and the selection of button 303 will result in variations occurring with respect to y directional movement. The selection of button 304 effectively cancels the operation of buttons 301 to 303 and ensures that the intensity of modifications applied to the existing image remains constant, irrespective of input modifications.
A similar situation is provided for adjusting the size of brush stamps.
Thus, the activation of button 305 allows size to be modified with respect to pen pressure, the activation of button 306 allows size to be modified with respect to the x position, the activation of button 307 allows size to vary withrespect to the y position and the activation of button 308 ensures that size remains constant irrespective of these variations.
Thirdly, the rate at which stamps are applied to the image may be modified with respect to pressure by the activation of button 309, with variations in the x and y direction being controlled by the activation of buttons 310 and 311. Again, the activation of button 312 ensures that the rate of stamp application remains constant irrespective of input variables.
The application of texture to brush stamps introduces noise in accordance with the present invention. When button 313 is activated, noise will be introduced into the brush stamping from a tiled texture map, thereby mitigating the undesirable effects due to the scaling of brush profiles. Button 313 is configured as a toggle and therefore once activated may be de-activated by a similar selection.
Canvas 201 is illustrated in Figure 4, in which a first stroke 401 has been created by a plurality of textured stamps 402. The size of these stamps may be increased to produce a bnush stamp of substantially similar profile but with a larger area of application. Thus, such a process may be implemented in order to create stroke 403 constructed from brush stamps 404 scaled up CA 0224404~ l998-07-28 from the brush stamps used to create stroke 401.
Each brush stamp is derived as a texture which is rendered into a frame buffer at sub-pixel definition in response to the position of the stylus 103 upon the touch tablet 104. Brush profiles are retained in memory in the form of a pixel patch which may consist of a square of 16 x 16 pixels, 32 x 32 pixels, 64 x 64 pixels, or 128 x 128 pixels, etc. depending upon the definition of images being processed. A patch 501 of this type is illustrated in Figure 5, consisting of an array of 16 x 16 pixel values. In this embodiment, when Iarger brushes are required of substantially similar profile, pixel patch 501 isexpanded to provide the number of pixels required.
The patch 501 represents the two dimensional area of coverage in terms of pixel locations and the profile of the brush stamp is defined by the intensity values of the pixels, which may be represented as a surface extending vertically away from the plane. Such a surface is illustrated as a projection 502, which provides a basis for calculating actual pixel values contained within the patch. Thus, although the profile may be visualised as a smooth surface, given the digitised nature of the patch, the values actually contained within the patch have discrete values therefore the actual profile may be considered as that represented at 502.
Profile 502 represents the profile of relevant pixels laying along line 503. The pixel at location 504 is set to its maximum intensity identified as 505. A similar intensity is allocated to pixels 506, 507 and 508. As pixel locations extend radially away from the centre, their intensities decrease.
Thus, pixel locations 509, 510, 511, 512, 513, 514 and 515 have intensity values represented at 516, 517, 518, 519, 520, 521 and 522 respectively.
Other profile shapes are stored representing different types of brushes, each with a pixel patch of 16 x 16 pixels. Generating a brush stamp which is four times larger in both the x and y dimensions, the 16 pixel x 16 pixel patch is scaled up to a 64 pixel x 64 pixel patch. Each pixel patch CA 0224404~ l998-07-28 contains fewer pixels than a full image frame and is arranged to be scaled to provide the size of patch required. Thus, a value from each pixel in the original 16 x 16 profile will be copied to 4 x 4 pixels in the scaled 64 x 64 pixel patch.
In addition to storing a patch for each brush profile, a further patch is stored representing substantially random noise. A new texture stamp is created for being rendered within the image frame. The first stamping texture, representing the profile, is scaled which may result in the creation of undesirable artefacts. However, the second stamping texture, representing the substantially random noise, is expanded without scaling, preferably by a process of tiling. The scaled profile is then combined with the tiled noise to produce the new texture map in a way such that the tiled noise mitigates the effects of the undesirable artefacts introduced by the scaling of the profile.
Part of a scaled brush profile is illustrated in Figure 6. Each small square, such as square 601, represents an individual pixel of the scaled texture. Each 4 pixel x 4 pixel square, identified by thickened boundaries 602, represents the regions scaled from a single pixel contained within the original profile. Thus, in this example, it may be assumed that region 602 is obtained from pixel position 514, region 603 is obtained from region 513 and region 604 is obtained from pixel 512. Thus, all pixels 605 within regions 604 will have an intensity of 519. All pixels 606 within region 603 will have an intensity of 520 and all pixels 601 within regions 602 will have an intensity of 521.
The tiling of noise is illustrated in Figure 7. In this example, it is assumed that the base tile consists of 4 pixels x 4 pixels although in practice the base tile would tend to contain significantly more pixels. Each pixel 701 within a tile 702 has a substantially random value although the amplitude of these values will tend to be restrained within a relatively low range, such thatthe magnitude of the noise data is relatively small compared to the magnitude of the brush profile data. In this way, the noise may be considered as introducing a level of "dithering" to the brush profile and is not intended to substantially modify the brush profile.
Pixel 701 has a particular noise value with pixel 702 having a randomly different value and again a different value is given to pixel 703 and 5 SO on. On the next tile, pixels at similar positions have similar values, introduced by effectively reproducing the base tile in a tiling operation. Thus,pixel 705, of tile 706 has the same value as pixel location 701.
Thus, it can be seen that the way in which brush profiles are expanded, as a scaling operation illustrated in Figure 6, differs from the way 0 in which the noise is expanded, as a tiling operation, as illustrated in Figure 7.
In this way, distortion artefacts introduced by each individual operation of tiling and the scaling of profiles do not mutually reinforce each other when thetwo expanded textures are combined, so that the combining process reduces unwanted artefacts arising from profile scaling.
The overall procedure is illustrated in Figure 8. The process shows a situation in which a brush stamp is required which is larger than the stored profiles. At process 801 a brush profile, of the type shown in Figure 5, is ~ scaled in the way described with reference to Figure 6. Process 802 produces an expanded tiled random texture, invoking a process substantially 20 similar to that described with respect to Figure 7.
At process 803 the scaled brush texture from 801 and the tiled random texture from 802 are combined to produce a composite brush stamp.
The position of the brush stamp to sub-pixel definition is identified as 804 and the brush stamp is rendered with an image at 805 under the control 25 of a control value derived from 806, usually derived from stylus pressure.
After rendering the new image at step 805 to determine new pixel values and filtering the image to take account of sub-pixel definition, the two images are combined as a modification process at step 806.

Claims (20)

1. Image data modifying apparatus, comprising image storage means, image display means, processing means and a manually controllable input means, wherein said storage means is arranged to store a plurality of modifying textures in the form of stamps having fewer pixels than a full image frame;
and said processing means is configured to construct a new texture stamp by scaling a first stamping texture, expanding a second stamping texture without scaling and combining said scaled and said expanded textures.

2. Apparatus according to claim 1, wherein said first stamping texture represents the brush profile.

3. Apparatus according to claim 2, wherein said storage means is configures to store a plurality of brush profiles.

4. Apparatus according to claim 1, wherein said second stamping texture represents random noise.

5. Apparatus according to claim 4, wherein said random noise has a relatively low amplitude.

6. Apparatus according to claim 1, wherein a new texture is rendered into an image at sub-pixel definition.

7. A method of modifying image data, in which pixel values in an image are modified in response to manual operation of an input device, comprising steps of storing a first stamping texture, storing a second stamping texture and producing a new stamping texture by scaling said first stamping texture while expanding said second said second texture without scaling.

8. A method according to claim 7, wherein said first stamping texture represents a brush profile.

9. A method according to claim 8, wherein a plurality of brush profiles are stored.

10. A method according to claim 7, wherein said second stamping texture represents substantially random noise.

11. A method according to claim 10, wherein said random noise has a relatively low amplitude.

12. A method according to claim 7, wherein a new texture is rendered into an image at sub-pixel definition.

13. A computer-readable medium having computer-readable instructions executable by a computer such that said computer performs the steps of:
storing a first stamping texture in a first data field;
storing a second stamping texture in a second data field; and modifying image data by producing an output stamping texture in response to scaling said first stamping texture while expanding said second stamping texture without scaling.

14. A computer-readable medium having computer-executable instructions according to claim 13, wherein said first stamping texture represents a brush profile.

15. A computer-readable medium having computer-executable instructions according to claim 14, wherein a plurality of brush profiles are stored.

16. A computer-readable medium having computer-executable instructions according to claim 13, wherein said second stamping texture represents substantially random noise.

17. A computer-readable medium having computer-executable instructions according to claim 16, wherein said random noise has a relatively low amplitude.

18. A computer-readable medium having computer-executable instructions according to claim 13, wherein a new texture is rendered into an image at sub-pixel definition.

19. A computer-readable storage system having a plurality of data fields stored on said medium and representing a data structure, comprising a first data field representing a first stamping texture; and a second data field representing a second stamping texture, such that, in response to image data modifying instructions an output stamping texture is produced by scaling said first texture and expanding said second texture without scaling.

20. A storage system having a plurality of data fields according to claim 19, wherein said first stamping texture represents a brush profile.