DE69033777T2 - Electronic graphic system - Google Patents

Description

The invention relates to an electronic graphics system for the modification of image data.

An electronic graphics system is described in our British Patent No. 2,089,625 and the corresponding US Patent No. 4,514,818. This graphics system is designed to electronically simulate the painting or drawing of a picture. The system includes a touch panel and pen combination which allows a user to select from a range of representational drawing elements and a range of colours on a menu display for use in simulated painting or drawing. The pen is moved across the touch panel and circuitry in the pen/touch panel combination generates position signals representing the position of the pen on the panel. The selected element covers a sequence of pixels in the generated image, with new pixels being derived for each pixel in the sequence covered by the element as position signals are generated by the pen/touch panel combination. New pixels are derived by a processing circuit depending on the distribution of the sequence of the selected element, the selected color and the old value of the pixel. The pen is pressure-sensitive and generates values representing the pressure, which are also used by the processing circuit to derive new pixels.

Pixels are stored in a suitable memory, such as an image memory, and a sequence of pixels is selected from the When the user, who is normally inexperienced with computer-based systems, manipulates the pen on the touch panel, the image is read from memory and modified by the processor to form a new sequence of pixels which are written back into memory. This process is repeated continuously while the pen is manipulated on the touch panel. Furthermore, the memory is repeatedly read and the pixels are presented to a color monitor to make the formation of the image observable in essentially real time.

It is important that the user can view his work in real time, i.e. without any perceptible delay between the movement of the pen and the resulting image on the monitor. This is achieved in our original "PAINTBOX" (registered trademark) system by processing all pixel sequences generated in a monitor frame period and writing the processed sequences back to memory within a frame period, at least when the time is averaged over several frame periods. However, in certain cases, such as a signature, which is naturally written quickly, operations in the system may be delayed compared to operations by the user, which is annoying to the user.

The present invention seeks to provide an improved electronic graphics system.

In our "PAINTPOX" system referred to above, options, features and modes of operation are selected from a menu displayed on the monitor. The system is designed so that the user can cause the menu to be displayed by quickly moving the pen over the edge or either side of the touch panel work area. Once the menu is displayed, the user can select from the available options by moving the cursor over the desired option. Some features are simply selected as ON or OFF by firmly pressing down with the pen while the cursor is over the desired feature, the system responding by changing the feature between ON and OFF accordingly. Other Features are selected as a numerical value within a predetermined range of values, for example from 0% to 100%, in previously known systems these features are selected by first moving the cursor over the desired feature and firmly pressing down with the stylus, whereupon the desired value is entered either via a separate keyboard or an equivalent of a keyboard displayed on the menu. The need to use a separate means of entering the numerical value slows down the use of the system and hinders the user.

EP-A-226959 describes a make-up simulator comprising a graphic image memory and a graphic image memory control device for controlling the graphic image memory, and a coordinate input device such as a tablet and a pressure-sensitive pen for inputting coordinates of a working position and a working condition of a graphic image of a face. A graphic image processing circuit performs necessary processing of a graphic image and controls a color television set.

The present invention provides an electronic graphics system for modifying image data stored in a memory and representing a plurality of image elements which together form an image, the system comprising: a display screen for displaying in an area thereof a menu containing representations of user-alterable features associated with modifications to the image and for displaying at least a portion of the image represented by the stored image data; cursor generating means for generating data for displaying a cursor on the display screen, the cursor generating means operable to select a feature for modification when the cursor is placed over the representation of the selected feature; and processing means responsive to selection of a feature by the cursor generating means for changing the selected feature in response to subsequent movement of the cursor on the display screen in an area separate from the representation of the selected feature.

The foregoing and other features of the invention are set forth with particularity in the appended claims and, together with advantages, will become more apparent from the following detailed description of exemplary embodiments of the invention when taken in conjunction with the accompanying drawings.

Short description of the drawings

Fig. 1 shows a video graphics system.

Fig. 2 shows a block diagram of processing circuitry used in the system of Fig. 1.

Fig. 3 shows a timing diagram showing the timing of the operations performed by the processing circuitry of Fig. 2.

Fig. 4 shows a schematic diagram of the system configured to allow modification of features presented in the menu.

Detailed description of an embodiment of the invention

As shown in Figures 1 and 2 of the accompanying drawings, a video graphics system 10 includes a processing and storage unit 15, a color (television type) display monitor 16, a touch panel 17, and a stylus 18. As will become more apparent from the following description, the processing unit 15 includes a disk drive for storing image data in programs, dynamic random access memory arrays (DRAMs) for storing image data, a service processor, a brush processor, filters, and special effects processors.

A memory array in the unit 15 stores digitized pixel data for a full video frame, which is read at full video frame rate to generate a video signal for the monitor 16. Furthermore, the data held in this memory can be modified in response to manual manipulation of the pen 18 on the touch panel 17. The pen 18 radiates energy detected by the touch panel, whereby circuitry in the touch panel determines whether the pen 18 is in proximity, ie at a predefined distance from the touch panel 17. When the pen 18 is brought into proximity, a signal is generated which identifies the position of the pen with a resolution which is greater than the pixel pitch of the video image. When the pen is in proximity, a cursor (in the form of a cross) is displayed on the monitor which follows the movements of the pen on the touch panel.

The pen 18 is also pressure sensitive so that a pressure value is generated by circuitry in the pen when it is brought into contact with the touch panel. This pressure value is applied to circuitry in the touch panel by radiant energy which is modulated by a signal representing a measure of the pen pressure. A combination touch panel and pen operating in such a wireless mode is manufactured by Wacom KK in Japan. As a function of the modulated radiation from the pen 18, the touch panel 17 thus provides signals to unit 15 indicating firstly the approach of the pen, secondly its x and y coordinates in the working area of the touch panel and thirdly the manual pressure exerted on the pen by the operator; all of this takes place without physical connection to the pen. It should be noted that in the pen/board combination described above, the pen itself does not contain an energy source, but reflects radiation from the board.

In addition to displaying a full screen of video, the monitor 16 can also display menus which are called up by striking the pen over the edge of the touch panel work area. A number of menus are available depending on the mode of operation of the system. From one of the available menus, a painting operation can be selected in which the movement of the pen 18 over the touch panel 17 simulates the effect of conventional artists' tools such as pencils, paint brushes, splatter brushes and chalk. Each item can be selected by moving the pen over the touch panel so that the cursor is displayed on a box in the menu which indicates the required characteristic is identified, after which pressure is applied to the pen. Once painting is selected in this way, together with a brush size and color, a stroke can be drawn with the pen, thereby generating a sequence of coordinate location and brush pressure signals. A service processor in unit 15 recreates the stroke from these coordinate values and modifies equally objectionable but overlapping circular representations of pixels in the memory array (image memory) for the displayed image.

In the following description, headings are used to identify the beginning of the description of the respective feature in order to help understand the features of the embodiment.

Pipeline processing

The process of repeated reading-modifying-writing in a frame store is described in our above-referenced British Patent No. 2,089,625 and the corresponding US Patent No. 4,514,818 of the applicant. In this process, a new pixel value (P new) is written into each pixel location in a mark which is derived from a previous pixel value (V old), a control value k determined by the shape of the selected element and a new color and intensity value (V new) according to the following equation:

Pneu = k · Vneu+(1 -k)Valt

In areas where the marks overlap, the pixel values (V old) have been previously derived from this equation, resulting in a repeated modification of the stored value towards the brush color, which inherently results in anti-pseudolines.

A corresponding procedure is chosen in system 10. In this system 10, however, the control value k is related to the pressure value resulting from the wireless connection between the pen and the touch panel. It should be noted that even if a position signal is generated by the touch panel when the pen is in proximity (as explained above), pixels are not modified until pressure is applied to the pin because k = 0 when no pressure is applied to the pin.

Our classic "PAINTBOX" (registered trademark) embodiment is fast due to its "real-time" processing capability. However, in certain cases the system may lag behind the artist's work. Such a lag may occur, for example, if the artist has selected an element covering a large area of pixels and is moving the pen rapidly across the touch panel. In these cases the number of pixels to be processed in a given period of time may be greater than what previously known systems can process in that time.

To improve the processing capability of the system and thereby reduce the observable delay between the artist's movement of the pen and the result of that movement appearing in the displayed image, the system 10 includes a pipeline processing arrangement. As will be apparent from the following description, the pipeline processing arrangement improves the processing capability by accessing pixels and performing a read-modify-write process in sequences of eight pixels to the image data in memory in response to user manipulations of the pen. Pipeline processing is accomplished by means of a buffer arrangement so that a first sequence can be processed while a second sequence is undergoing a memory transfer.

Referring to Figure 2 of the accompanying drawings, the processor unit 15 of Figure 1 includes a processor circuit, generally designated 10, which includes an image memory 21, which may be, for example, a dynamic random access memory (DRAM) device, and a brush processor 22. A sequence of pixel data stored in the image memory 21 is read into a buffer 23, and, as will be described in more detail below, individual pixels in the sequence are selected by a selector 25 under the control of a read counter 24 for input to the brush processor 22.

The system 10 further includes a color memory 11 for storing data representing a color selected by a user and input to the processor 22 via a "color" line in parallel with the pixel input data from the read address selector 25. When the pen 18 is near the touch panel 17, the touch panel 17 generates XY coordinate data representing the instantaneous position of the pen on the touch panel. The XY coordinate data is input to a sequence address generator 12 which converts the XY coordinate data into a sequence of addresses used to address a brush shape memory 13 and a pen pressure memory 14. The brush shape memory holds data defining a three-dimensional shape representing a selected character element, while the pen pressure memory 14 holds instantaneous pen pressure values from the touch panel and the pen. Data from the brush shape memory 13 and from the pen print memory 14 are entered into the processor 22 via the FORM and PRINT lines, respectively.

The pixels input to processor 22 from read address selector 25 are modified by and output from brush processor 22 to be written back to frame buffer 21 via write address selector 28 under control of write counter 26 and buffer 27. Write address selector 28 and buffer 27 perform an inverse function with respect to buffer 23 and read address selector 25 in that write address selector 28 receives modified pixels from brush processor 22 and writes those pixels to buffer 27 which supplies sequences of pixels to frame buffer 21.

It should be noted that the processor circuit 10 includes control and timing circuits for monitoring the transfer of pixels in the circuit. For clarity, the control and timing circuits are not shown in Figure 2. Clock signals are used to control the operation of the processor circuit 10; a description follows of the operation of the processor circuit 10 in which pixel data transfers occur during specific clock periods. It should be noted that these clock periods are defined by the control and timing circuits mentioned above.

Fig. 3 of the accompanying drawings shows a timing diagram of the operation of the processor circuit 10.

The period between each vertical line (a, b, c...k, l) in the timing diagram represents a group of four clock pulses of 74 ns (13.5 MHz). The periods each represent a time interval in which four pixels are required to produce an output signal for display on the monitor 16 at video frequency. To prevent the monitor 16 from continuously requesting access to the frame buffer 21 and to prevent access to the frame buffer 21 by the brush processor 22 during alternating groups of four clock periods, for example 31-1 to 314, eight pixels are read from the frame buffer 21 for the monitor 16 via a buffer 16a (this function is identified in Fig. 3 by the word "VIDEO" on the line labeled "FRAME BATTERY"). This solution makes alternating groups of four clock periods available for data transfer between the frame buffer 21 and the brush processor 22. In this way, the reading of the contents of the frame buffer 21 by the monitor 16 and the brush processor 22 is interleaved between alternating periods. In Fig. 3, eleven periods (3141) of four clock cycles are identified.

The operation of the processor circuit is as follows. During the first period 31, eight data pixels are read out in parallel from the image memory 21 and entered into the addressable buffer 23. This operation is represented by the designation READ 1. During the period 32, after a clock period 32-1, the read counter 24 supplies individual addresses to the read address selector 25, which selects and enters individual pixels for the brush processor 24 at the clock frequency. The read counter 24 and the read address selector 25 are designed in such a way that after a sequence of eight pixels for the buffer 23 has been read out once, the pixels are read out individually in sequence and supplied to the brush processor 22 by the read address selector 25. The read counter 24 counts cyclically through a read sequence to control the read address selector 25.

The brush processor 24 is designed as a pipeline with eight stages. To ensure that the various input data for the brush processor 22 arrive at the correct location in the processor at the correct time, each data transfer path in the processor is designed to include eight delay stages between the input and output of the processor. The delay stages are implemented either by a latch (L1 to L8) with a single delay period or by a multiplier (M1, M2) with two delay periods.

Pixels are input serially from buffer 23 through read address selector 25 to brush processor 22. As each pixel is input to latch L1, pixels previously input there move through the pipeline of latches such that the pixel previously input to latch L1 is moved to latch L2, and continues accordingly. At the same time, data representing the color and shape selected by the user, along with data representing the pressure applied to the pen, are input to the brush processor in parallel on lines COLOR, SHAPE and PRESSURE, respectively. As a pixel is clocked through the brush processor, it is subtractively combined with the corresponding color data by subtractor 50, while the shape data and pressure data are multiplied together by multiplier 51. The latches in the processor are arranged so that the sum B - A, where B is the color data and A is the input pixel data, and the product K, where K = shape x pressure, are input to a multiplier 52 during the fifth and sixth clock periods, the product being provided by the multiplier 52 for addition to the original pixel data A. The resulting sum represented by A + Q in Fig. 3

A+Q = (1-K)A+KB

is output as a new pixel from the processor 22 via the latch L8 during the eighth clock period. Thus, pixel data input to the processor is modified by the processor 22 and output therefrom eight clock periods later for feed back to the frame buffer 21. It should be noted that the new pixel is therefore derived using the same form of algorithm as described above, i.e.

A+Q = KB+(1-K)A

is of the same form as the equation discussed above

Pneu = K·Vneu+(1-K)Valt.

Referring to Fig. 3, during period 32 eight pixel values are read out as video from frame buffer 21 and buffered in buffer 16a to produce a continuous video signal for monitor 16. Period 33 is used to write previously processed output data from buffer 27 back to frame buffer 21. During period 33 read counter 25 continues to clock pixels into brush processor 22, this function being represented in Fig. 3 by the designation INPUT PIXEL SET 1. In period 34 video access is again required, again outputting data as video to buffer 16a for display on monitor 16. After the first clock cycle 34-1 of period 34 all pixels in the input pixel set 1 function have been read out of buffer 23 and the first output pixel value is thus available at latch L8. Pixel values are output from latch L8 and written into a buffer 27 via write address selector 28 under control of write counter 26. Write counter 26 operates in a similar manner to read counter 24 in that it cyclically counts through a write address sequence for buffer 27. The function of reading pixels for write counter 26 is represented in Fig. 3 by the designation output pixel set 1; this function occurs during the period between the end of clock period 34-2 and the end of clock period 36-2.

At the beginning of period 35, during the output pixel set 1 function, two output values are clocked into buffer 27 so that latches L1 and L2 are empty. During period 35, no video access is required because during the previous period the video for this period was read into buffer 16 so that another eight pixels are read from frame store 21 and input into buffer 23; this function is represented in Fig. 2 by the designation READ 2. During period 36, a video access occurs and reading from the buffer 23 and input to brush processor 24; this function is represented by the designation INPUT PIXEL SET 2. At the end of clock cycle 36-2, all eight outputs from the output set 1 function have been clocked into buffer 27. During period 37, the pixels of input pixel set 2 continue to be clocked into and through image processor 22 while the eight output pixels of output pixel set 1 are written into frame buffer 21 (WRITE 1). During period 28, a video access occurs and the last pixel of input pixel set 2 from the second read (READ 2) is provided to the brush processor while the first two outputs of output pixel set 2 are provided to buffer 27. Period 39 corresponds to period 31, with READ 3 being processed in the same manner as READ 1, with the entire process being repeated for additional pixels from frame buffer 21.

The system 10 thus represents an arrangement which interleaves the reading of pixel data for modification with the reading of pixel data for display so that the effective operating speed of the system is increased.

Pen above the image to control box values

As mentioned above, modes of operation can be selected from a menu invoked by the user by striking the pen over the edge of the work area, i.e., rapidly moving the pen over the edge of the touch tablet. Certain boxes of the menu allow the selection of features or functions of the system in an ON-OFF manner, but must be substituted for other numerical values. In our previous systems, these values were defined by the artist using either a physical keyboard or its equivalent defined in the menu area. This results in slowing down and hindering the natural movements of the artist operating the system. The present system seeks to eliminate this problem by allowing the artist to control menu sizes by moving the pen over the display, and preferably over the image area, i.e., outside the menu area, during a menu feature selection operation.

To accomplish this, as shown in Figure 4, a service processor 70 compares successive coordinate values and translates these values to produce numerical increments. First, the artist selects the required menu, e.g., menu 71, which is displayed on a portion of the screen of monitor 16. The artist then moves the pen 18 across the touch panel 17 until a cursor 72 is over a box 73 in the menu 71. The box 73 represents a desired feature or function to be changed and displays a numerical value (not shown) representing the current value of that function. With the cursor 72 over the selected box 73, the artist then applies pressure to the pen, i.e., the artist applies pressure to the pen. The service processor 70 responds by selecting the desired function or feature from a memory 74 and informs the operator that the box 73 has been selected by changing the color, for example from pink to green. The processor 70 is now ready to respond to movements of the pen. The artist can move the pen 72, for example in the pen and touch tablet combination 17, 18, from the menu 71 to an image area 75 of the display, with further movement of the cursor 72 on the display screen being interpreted by the processor 70 as a command to increase or decrease the value of the selected feature. For example, a movement to the left L can be interpreted as a command to decrease the value, while a movement to the right R can be interpreted as a command to increase the value. Thus, by moving the pen 18 so that the cursor 72 is outside the menu area and by exerting pressure on the pen and moving it, the numerical value is decreased with a movement to the left and increased with a movement to the right.

Once the desired value has been obtained, the cursor 72 is again moved in the menu area 71 and pressed with the cursor placed over the selected box. The processor 70 interprets this action as a command to stop the setting of the selected feature and to store the new value in the memory 74. The The feature is then deselected by the processor 70, indicating this by changing the color of the selected box back to its original color. Additional features may then be selected by the artist if required, or the menu may be removed from the display, returning the system to a drawing mode by striking the pen over the edge of the work area. Note that once a menu has been invoked, normal painting in the image area is inhibited, so adjusting parameters in the manner described above will not cause unwanted lines to be placed in the image.

Although the present invention has been described with reference to a preferred embodiment, it is to be understood that the embodiment in question is merely exemplary.

Claims (7)

1. An electronic graphics system for modifying image data stored in a memory and representing a plurality of image elements that together form an image, the system comprising: a display screen (16) for displaying in a region thereof a menu (71) containing representations of user-changeable features (73) associated with the modifications to the image and for displaying at least a portion (75) of the image represented by the stored image data; cursor generating means (17, 18) for generating data for displaying a cursor (72) on the display screen, the cursor generating means being operable to select a feature for change when the cursor is placed over the representation of the selected feature; and processing means (70, 74) responsive to selection of a feature by the cursor generating means for changing the selected feature in response to subsequent movement of the cursor on the display screen in an area separate from the representation of the selected feature. 2. An electronic graphics system as claimed in claim 1, wherein the cursor generating means comprises a combination of a pen (18) and a touch panel (17) and is arranged such that, in use, movement of the pen on the touch panel causes the cursor (72) to move on the display screen (16) and the application of pressure to the pen when the cursor is placed over a representation of a desired feature causes selection of the desired feature. 3. Electronic graphics system according to claim 1 or 2, wherein the combination of pen (18) and touch panel (17) is arranged to produce coordinate data (x, y) representing the position of the pen on the touch panel and pressure data representing the pressure exerted on the pen, the processing means (70, 74) being arranged to receive the coordinate data and the pressure data and to modify the value of a selected feature in response to the received data. 4. An electronic graphics system according to any one of claims 1 to 3, wherein once a feature has been selected, the processing means (70, 74) is configured to respond to subsequent movement of the cursor (72) in one direction by increasing the value of the selected feature and to respond to subsequent movement in another direction by decreasing the value of the selected feature. 5. Electronic graphics system according to any one of the preceding claims, wherein the processing means comprises a processor (70) and a memory (74) for storing data representing the user-changeable features (73) contained in the menu (71). 6. An electronic graphics system according to any preceding claim, wherein the processing means (70, 74) is arranged to change the selected feature in response to subsequent movement of the cursor (72) over the portion of the image displayed on the display screen. 7. An electronic graphics system according to any preceding claim, wherein the cursor generating means (17, 18) is operable to deselect a selected feature when the cursor is again placed over the representation of the selected feature, the processing means (70, 74) being arranged to respond to the de-selection of the feature by storing the changed data representing the selected feature.