EP1365359A1 - Procédé et système de traitement d'images - Google Patents

Procédé et système de traitement d'images Download PDF

Info

Publication number
EP1365359A1
EP1365359A1 EP02253679A EP02253679A EP1365359A1 EP 1365359 A1 EP1365359 A1 EP 1365359A1 EP 02253679 A EP02253679 A EP 02253679A EP 02253679 A EP02253679 A EP 02253679A EP 1365359 A1 EP1365359 A1 EP 1365359A1
Authority
EP
European Patent Office
Prior art keywords
image
shape
sequence
images
cylinder
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP02253679A
Other languages
German (de)
English (en)
Inventor
designation of the inventor has not yet been filed The
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
British Telecommunications PLC
Original Assignee
British Telecommunications PLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by British Telecommunications PLC filed Critical British Telecommunications PLC
Priority to EP02253679A priority Critical patent/EP1365359A1/fr
Priority to CNA2008101489274A priority patent/CN101447093A/zh
Priority to PCT/GB2003/001961 priority patent/WO2003100728A2/fr
Priority to JP2004508100A priority patent/JP4354907B2/ja
Priority to CA002484491A priority patent/CA2484491A1/fr
Priority to CNA038112876A priority patent/CN1653490A/zh
Priority to EP03725373A priority patent/EP1508126A2/fr
Priority to KR1020047018978A priority patent/KR100826443B1/ko
Priority to AU2003227910A priority patent/AU2003227910A1/en
Priority to US10/513,032 priority patent/US7184049B2/en
Publication of EP1365359A1 publication Critical patent/EP1365359A1/fr
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings

Definitions

  • the present invention relates to an image processing method and system, to a computer program arranged to cause a computer to perform the method, and to a computer readable storage medium storing said computer program.
  • the delivery of synthesised or recorded speech messages can be enhanced by the use of an animated picture of the sender, or by displaying at least the head part of an avatar created of the sender, but in both cases in which only the lips move in synchrony with the reproduced speech.
  • a picture of the sender is used, the impression of movement of the lips is created by displaying what is known as a "viseme", which is an image of a human face (for example of the message sender) provided with the lips thereof in one of a number of identifiable shapes, which each represent a lip shape associated with one or more phonemes.
  • Phonemes are, of course, well known in the art and are the individual discrete sounds which are used within a language. It is estimated that there are approximately 44 phonemes in the English language, but perhaps only as few as twenty or so visemes. Therefore, it is possible to display the same viseme when reproducing one of several phonemes.
  • a speech reproducer such as a speech synthesiser outputs acoustic waveforms corresponding to a sequence of phonemes, and at the same time a display means displays to the user the appropriate viseme associated with the particular phoneme which has been reproduced at any particular time. Therefore, the user obtains the illusion of an image of the sender whose lips appear to move in synchrony with the reproduced speech. It should be noted that here the visemes are two dimensional images of the sender.
  • the alternative method known in the prior art is to produce either a whole body avatar, or at least a three dimensional virtual model of the sender's head, which is then shaped and textured to look like the sender.
  • the lips of the head model can then be controlled to move in synchrony with the reproduced speech, such that the lips of the model assume the appropriate shape for the particular phoneme being reproduced at any particular time.
  • Such systems involve complex head modelling using a virtual wire frame reshaped by difficult image processing or invasive sensing, and requires a process in which a still picture is accurately conformed to the given model.
  • the first of the aforementioned methods being that of displaying a sequence of two-dimensional visemes in synchrony with the reproduced speech, does not suffer from the same computational intensity problems as the second of the aforementioned methods, but does suffer from the problem that the displayed image appears to be almost robotic to the viewer, in that they can appear stale, automated, and not life-like. This is because the only movement apparent to the viewer is the movement of the lips to create the appropriate viseme shape corresponding to the present phoneme being reproduced.
  • the present invention addresses the above problem by providing an image processing method and system which is able to process the two-dimensional viseme images in order to produce processed images which when displayed in sequence reproduced the observed small movements of a human head during speech.
  • the image processing is achieved conceptually by texturing the head image to be processed onto the surface of a 3D shape, which is preferably a 3D virtual shape provided within a virtual space, and then moving the shape slightly to imitate the observed human head movements. Once the shape has been moved slightly a projection of the image from the surface of the shape back to the original image position is taken, which results in a second, processed, image, which is an image of a human head with a slight aspect ratio change.
  • a sequence of viseme images are processed by the method in turn, and the resultant processed images subsequently displayed to a user in turn, the result is that the observed random movements of a human head during speech are simulated.
  • an image processing method comprising the steps of:
  • the present invention provides the advantage that an effective image processing operation which reproduces a three dimensional aspect change of the subject displayed in the image can be simulated. There is a further advantage that as both the input and the output are two dimensional images, then computational intensity of an algorithm embodying the method is reduced.
  • the first image forms part of a sequence of first images, the method further comprising repeating steps (a), (b), (c) for each first image in said sequence to obtain a corresponding sequence of second images.
  • the invention can be applied to a sequence of images in turn in order to allow the same processing to be applied to an "animation" sequence.
  • the respective sequences of first and second images preferably each form an animated sequence of a human head speaking.
  • the moving step further comprises randomly moving the shape and/or the image position. This provides the advantage that when the images are of human heads, the observed human movements which the preferred embodiment of the present invention is attempting to recreate are more accurately reproduced.
  • the movement comprises rotating the three dimensional shape of about one or more axes thereof. This provides the advantage that, when the images are images of human heads, the movement by the shape simulates the movement possible of a human head attached to a pair of shoulders.
  • the present invention further provides an image processing system comprising:
  • the second aspect possesses the same further features and advantages as previously described in respect of the first aspect.
  • a computer program arranged such that when executed on a computer it causes the computer to perform the method of the first aspect of the invention.
  • a computer readable storage medium storing a computer program according to the third aspect.
  • the computer readable storage medium may be any magnetic, optical, magneto-optical, solid-state, or other storage medium known in the art, for example a hard disk, a portable disk, a CD Rom, a DVD, RAM, ROM, programmable ROM, tape, or the like. It should be noted that the list of computer readable storage media given above is not exhaustive, and any known computer readable storage media may suffice.
  • the first embodiment of the present invention provides an image processing method and system which provides for the processing of a first image 62 in order to provide a second, processed, image 72.
  • image processing performed by the present invention is illustrated in Figures 6 and 7 and can be described as follows.
  • the first image to be processed, image 62 is provided at a certain position and orientation with respect to a three dimensional shape, being in this case the cylinder 60.
  • the cylinder 60 is a three dimensional virtual shape provided within a virtual space, and the image is provided at the certain position and orientation located within the virtual space.
  • the image is provided located with respect to the cylinder 60 such that the plane of the image is parallel to the axis of the cylinder.
  • the orientation of the image is such that the "up and down" axis of the image as shown is also parallel to the axis of the cylinder 60.
  • the cylinder 60 is provided with a curved outer service 64.
  • the image 62 is "textured" onto the surface 64 of the cylinder by projecting the image 62 onto the surface 64 from the image position.
  • the image orientation is maintained but that the image aspect ratio when applied to the outer surface 64 is changed slightly due to the curving nature of the surface of the cylinder.
  • texturing we mean that the individual pixel luminance and chrominance values are applied to the surface 64 of the cylinder 60 in accordance with the projection of the image thereonto, so that it appears as if the surface 64 is "painted” with the projection of the image 62.
  • the texturing of the surface 64 with the projection of the image 62 effectively binds the respective pixel luminance and chrominance values to the surface 64, such that the image appears to be affixed thereto.
  • the surface 64 is fixedly textured with the projection of the image 62, such that even if the cylinder 60 is moved in any way, the image texturing on the surface 64 moves therewith.
  • the next step in the image processing method is that the cylinder 60 is moved slightly, in this case by a rotation about its axis to a second position 60', as shown in Figure 7. Because the projection of the image 62 onto the surface 64 is textured thereon such that it is bound thereto, the textured image on the surface 64 also rotates with the cylinder to a second position 64' as shown in Figure 7.
  • the movement of the cylinder 60 to the second position 60' may be any movement necessary to obtain the desired effect.
  • the movement is preferably a rotation about the axis of the cylinder of less than ten degrees in either direction, and preferably of no more than one degree.
  • the rotation may be in either direction clockwise or anti-clockwise about the axis.
  • the movement of the shape i.e. in this case the cylinder 60
  • the shape is a cylinder and the images are images of heads
  • the movement of the shape is a rotation about the axis thereof of no more than ten degrees in either direction, but preferably of no more than one degree, the amount of rotation and the direction being randomly chosen within these limits.
  • the final step in the image processing method of the embodiments of the invention is to take a projection of the pixel luminance and chrominance values bound to the textured surface 64' from the surface back to the position in relative space of the first image 62.
  • the shape has been moved it is necessary to take the projection from the surface back to the original position of the first image in order that the effect of the movement in producing an aspect ratio change of the image is achieved. If the textured surface were merely projected in a direction substantially orthogonal thereto, then the resultant image would be identical to the first image 62.
  • a second, processed, image 72 formed from the pixel luminance and chrominance values as projected to the first image position is obtained in the same position and orientation as the first image 62, but with the content thereof processed slightly to represent an aspect change due to the movement of the cylinder 60.
  • the movement that is applied to the shape is preferably randomly chosen, and where the shape is a cylinder is preferably a rotation about the main axis of about no more than one degree.
  • Such movements of the shape when textured with each image simulates the natural movements of a human head when speaking.
  • the movement may be linked to a measure of the energy in the speech which is to be reproduced at the same time as the resultant processed image 72. For example, where the energy in the speech is great (i.e. the speech is loud) the movement of the shape may be greater than where the energy in the speech is relatively low. This will give the effect of the head presented in the image moving a greater amount when the reproduced speech is of a louder volume.
  • Another approach is to control the shape movement via waveform analysis, such that the movement is determined in accordance with the frequencies included within the speech.
  • the speech waveform is subject to Fourier analysis to determine the frequencies thereof, and the movement of the shape is controlled in dependence on the found frequencies.
  • the shape movement can be controlled so as to move the shape more if there is more energy in the lower frequencies of the speech than the higher frequencies, or vice versa.
  • control of the movement is determined not simply upon the sound being reproduced simultaneously, but instead the shape may be moved both before and after sound reproduction is taking place.
  • the shape can be moved to the largest degree (say from -8 degrees to +8 degrees) from side to side quickly, and then once the loud and quick speech has finished the side to side movement of the shape is gradually reduced and slowed down back into its neutral state.
  • the movement of the shape would continue to occur in a diminishing manner even after the speech has stopped.
  • Figure 5 illustrates various shapes which may be used with the present invention. More particularly, Figure 5a depicts a sphere, which may be used as the shape. In such a case the sphere may be rotated in any direction, and not just about its polar axis. It is also possible for plural rotations about any access thereof to be applied sequentially to achieve more complex movements.
  • Figure 5b illustrates a cylinder and preferably the movement applied thereto is a rotation about the long axis, as previously described. Other rotations may be possible, however.
  • Figure 5c illustrates an ellipsoid.
  • the movements which may be applied to the an ellipsoid of Figure 5c are similar to those which may be applied to the sphere shown in Figure 5a, that is, a rotation about the polar axis, but also about any other axis thereof.
  • Figure 5d shows an ovoid shape.
  • the shape of Figure 5d most closely represents that of a human head and hence may produce advantageous effects in its use.
  • the movements which may be applied to the shape of Figure 5d are the same as those which may be applied to the sphere or the ellipsoid, that is a rotation about any axis thereof, but preferably about the polar axis.
  • Figure 5e shows a shape which we have termed a "double cylinder” that is, a cylinder the upper and lower surfaces of which are shaped as sections of a second cylinder orthogonal to the direction of the first cylinder. Any movement may be applied to this shape, and in particular any rotation about any axis thereof.
  • the textured shape it is not essential that the textured shape be moved at all; what is essential is that there is a relative movement of the shape with respect to the image position to which the projection of the textured surface is to be made i.e. the image position at which the second, processed image is obtained.
  • the shape may be kept in the same position and the image position to which the projection of the textured surface is made moved around the shape.
  • both the image position and the shape may be moved, the respective movements being different to give a relative movement therebetween.
  • the movements which may be applied to the image are the corollary of those described above for the shape, e.g. a rotation about one or more axes of the shape (note not of the image).
  • FIG 14 illustrates a mobile telephone according to the second embodiment of the present invention. More particularly, a mobile telephone 100 is provided, which is itself provided with a display screen 102 and a keypad 104 to allow data entry by the user. An audio reproducing means 108, being an audio transducer, is further provided. When a text message is received at the mobile telephone 100 an image 106 of the sender of the text message is displayed on the screen 102, the image being a sequence of visemes which are chosen to match the phonemes being reproduced by the audio reproducing means 108 to articulate the received text message. In accordance with the second embodiment of the invention the image 106 has been processed so as to confer on the image the life like movements observed of a human when speaking. The processing of the image 106 to achieve this effect is described next.
  • Figure 1 illustrates a system block diagram of the necessary elements required to perform the image processing located within the mobile telephone 100. More particularly, within the mobile telephone 100 is provided a text message receiving means 10 which is arranged to receive a text message, for example in accordance with the short message system (SMS) protocol, and to deliver the text of the received text message to a text buffer 12, wherein it is stored. The text of the message stored in the buffer 12 is read by a parser 14 which acts to parse the text to determine the particular phonemes and the order of reproduction thereof which will be required in order to convert the text into a spoken output.
  • SMS short message system
  • the parser 14 therefore takes the text from the text buffer 12 as input, acoustically parses the text, and outputs a sequence of phonemes corresponding to the text to the control means 16.
  • the precise steps to be performed by the parser in parsing the text to produce the phoneme sequence are known in the art, and in particular with respect to prior art text-to-speech systems such as the BT Laureate system.
  • the representation of the phonemes passed to the control means 16 may be any standard phoneme representation, such as SAMPA or the like.
  • a storage medium 20 which provides a storage area for a phoneme store 210 wherein acoustic representations of phonemes are stored, such as in the form of waveforms stored as .WAV files.
  • the storage medium 20 further provides a viseme store 208, which stores a sequence of visemes, suitable for display on the display 102.
  • a viseme is a image of a human face with a particular lip shape associated with one or more phonemes.
  • a shape store 206 which stores shape data corresponding to any of the geometrical shapes such as spheres, cylinders, ellipsoids, ovoids, or double cylinders, as shown in Figure 5.
  • the data representing any particular shape preferably takes the form of the coordinates in three dimensional virtual space of the vertices of a large number of polygons which together make up the shape.
  • the shape surface is further defined by the shape data further comprising vertex connection information which specifies how to connect up the vertices to form polygons.
  • a set of vertex point coordinate information, and vertex connection information is stored for each possible shape, within the shape store 206.
  • an operating system program 202 which provides the necessary functionality and protocols for the control means 16 to be able to control and communicate with the various other system elements, as well as an animation program 204 which specifically controls the operation of the control means to control the various other elements to perform the present invention.
  • the storage medium 20 provided within the mobile phone 100 is a solid state storage medium, a FLASH multimedia card, or the like.
  • an image processor 18 which is controlled by the control means 16 in accordance with instructions contained within the animation program 204, and which is further arranged to access both the viseme store 208 to obtain viseme images therefrom, and the shape store 206 in order to obtain data representing 3D geometric shapes therefrom.
  • the image processor 18 is capable of reading the 3D shape data, in order to model the shape therein.
  • the image processor 18 is arranged to output processed images to the display means 102 for display thereon.
  • the audio reproducing means 108 are arranged to receive phonemes waveforms from the phoneme store 210 and to reproduce the phoneme waveforms in the received order to reproduce the speech represented thereby.
  • the phoneme store 210, the viseme store 208, and the image processor 18, are all under the control of the control means 16 which maintains synchronisation therebetween. More particularly, the control means 16 controls the phoneme store 210 to output phonemes to the audio reproducing means 108 in sequence to reproduce the received text message as speech. Similarly, the control means 16 controls the viseme store 208 to cause the viseme store 208 to output the correct viseme image to the image processor 18 corresponding to a particular phoneme to be reproduced which is to be output from the phoneme store 210.
  • the image processor 18 processes the image received from the viseme store and outputs the processed image to the display means 102.
  • Figure 2 illustrates the system functions performed by the image processor 18. More particularly, within the image processor 18 is provided a controller 8, which receives control signals from the control means 16. Control lines shown as dotted lines in Figure 2 are provided from the controller 8 to each of a texturing means 2, a shape modelling means 4, and a projector means 6.
  • the texturing means 2 receives at its input signals representing images output from the viseme store 208.
  • the shape modelling means 4 receives at its input the data representing the three dimensional shapes from the shape store 206.
  • the shape modelling means 4 is capable of interpreting the received representative data to virtually model the shape represented by the data, such as a cylinder, sphere, ellipsoid, ovoid, or the like.
  • a projector means 6 which receives information from the shape modelling means, and acts to perform a virtual projection of the textured surface of the shape to an image position to obtain a second processed image.
  • the projector means 6 therefore outputs the processed image from the processor 18.
  • Figure 3 represents a flow diagram of the operation of the overall system
  • Figure 4 is a flow diagram of the steps specifically performed by the image processor 18 in accordance with the present invention.
  • a text message is received by the text message receiver means 10.
  • the text message is stored in the text buffer 12, and then at step 3.6 the parser 14 acts to read the text message as required from the text buffer 12.
  • the parser 14 parses the text of the message to obtain the phoneme representation thereof at step 3.8. As discussed previously, parsing of text to obtain a phoneme representation is known in the art.
  • the phoneme representation obtained by the parser is passed to the control means 16, which at step 3.10 controls the phoneme store to read the acoustic waveform of the first phoneme in the sequence received from the parser therefrom.
  • the control means controls the viseme store to output from the viseme store 208 the corresponding viseme to the phoneme being presently output from the phoneme store 210.
  • the phoneme output from the phoneme store 210 is passed to the audio reproducing means 108, whereas the viseme output from the viseme store 208 is passed to the image processor 18.
  • the image processor 18 processes the received viseme in accordance with the present invention in order to "animate” the image by applying the image processing method of the present invention thereto. This produces the effect on the viseme as if the head represented in the viseme has moved slightly.
  • the processed viseme is output to the display means 102 where it is then displayed to the user.
  • the audio reproducing means 108 plays the acoustic waveform of the present phoneme as received from the phoneme store 210.
  • step 3.20 determines whether all the phonemes in the sequence determined by the parser 14 have been played. If the evaluation determines that all the phonemes have been played then the procedure ends. In contrast, if it is determined that there remain phonemes to be played, and associated visemes to be displayed, then at step 3.22 the controls means 16 moves onto the next phoneme in the sequence and processing returns to step 3.10.
  • the processing steps of step 3.10, step 3.12, step 3.14, step 3.16, step 3.18, step 3.20, and step 3.22 are then repeated in a loop until all of the phonemes output from the parser 14 to the control means 16 have been played, and the associated visemes processed and displayed to the user.
  • the image processor 18 receives a viseme output from the viseme store 208 under the control of the control means 16.
  • the image processor 18 accesses the shape store 206 and retrieves the data representative of the 3D shape to which the received image is to be applied.
  • the shape modelling means 4 provided in the image processor 18 receives the data from the shape store, and uses the data to model the shape as a three dimensional geometric shape in virtual space.
  • the texturing means acts at step 4.6 to texture the surface of the shape with a projection of the received image.
  • texturing we mean that the luminance and chrominance values of the image pixels as projected onto the surface of the shape are effectively bound to the surface, such that the surface is effectively "painted" with the image projection, as previously described.
  • the shape modelling means 4 acts to perform a movement of the shape and models the textured shape in the moved position relative to the image position.
  • the movement may be a rotation about any axis of the shape, and in the preferred embodiment the shape is a cylinder and the movement is a rotation about the main access thereof by no more than ten degrees, and preferably by no more than one degree in either direction.
  • the precise movement applied to the textured shape is preferably randomly chosen for each iteration of the process, with the result that each processed image has had a different shape movement applied thereto.
  • the movement of the shape may be controlled in response to the speech energy in the phoneme which is to be reproduced with the processed viseme.
  • it is the image position which is moved relative to the shape, or a combination of both shape and image position relative movement, as discussed previously.
  • it is the relative movement between the shape and image position of the processed image which is important, not the absolute movements of each.
  • the projector means 6 obtains information of the moved textured shape from the shape modelling means 4 and acts to take a projection of the textured surface of the shape to the position in virtual space of the original received image.
  • the image (comprising the projected pixel luminance and chrominance values) obtained by such a projection located at the position of the original received image is then output by the projector means as the output from the image processor 18 to the display means 102, at step 4.12.
  • the display means 102 then displays the processed image to the user, as described previously.
  • the present invention provides an image processing method and system which allows for images to be processed to produce a particular "aspect ratio change" effect by virtue of the image being applied to a three dimensional shape, a relative movement then being effected between the shape and an image position, and a projection of the applied image then being taken to the image position to obtain a second image.
  • the images are visemes
  • the relative movement of the shape with respect to the image position to achieve the aspect ratio change effect within the resultant image simulates the small head movements observed of real humans when talking. Therefore, the wooden, lifeless nature of previous viseme images is alleviated, and the overall effect is of a more lifelike image.
  • the invention is not limited to processing images of human heads, and images of animal heads, fantasy heads or the like may be used.
  • “fantasy heads” we mean images of the heads of fantasy creatures such as, for example, those of the TeletubbiesTM as shown on BBC Television, or those of characters from fantasy films such as Star Wars® or the like.
  • the invention has been specifically developed for processing images of faces (whether, human, animal, fantasy or otherwise), its use is not limited to the processing of facial images, and any image which requires an aspect ratio change as previously described and provided by the invention may be processed thereby.
  • the image is projected onto a shape, the shape then rotated, and then the projection taken from the shape back to the original image position.
  • the concept as already described may be maintained, and especially using specialist three-dimensional virtual reality programming languages as are known in the art, it should also be understood that the same image processing effects can be obtained via a purely mathematical algorithm. That is, the image to be textured onto the shape can be considered to be simply an array of points located at discrete coordinates in a known co-ordinate system, and the shape instead of being represented by polygons as in the preferred embodiment, may instead merely be represented by an appropriate equation in the known coordinate system.
  • the resultant processed image can then be mathematically obtained using the equation for the geometric shape, and applying the appropriate transform to the image coordinates using the shape equation.
  • Figure 8 shows a perspective view of the three-dimensional virtual space in which the image and shape objects of the present invention exist.
  • Figure 8 illustrates the respective positioning and orientation of the image to be processed and the virtual shape model within the virtual space.
  • Figure 9 illustrates the image to be processed from a viewpoint directly in front, i.e. orthogonal to the plane of the image.
  • Figure 10 shows a view taken from above in the direction of the axis of the cylinder. In this case the view is along the image plane, such that image is no longer evident from this view-point (due to it being two-dimensional).
  • Figures 11, 12, and 13 illustrate a similar view.
  • the present invention can be embodied in either the conceptual basis as described in respect of the preferred embodiment, or on a purely mathematical basis as shown above, and that the two are not mutually exclusive.
  • the same effect of obtaining a processed image which has undergone an "aspect ratio change" is obtained in either case.
  • an image projector arranged to display an image via projection onto a surface of a shape.
  • the image projector may be a slide projector, or a digital light projector, or the like.
  • a three-dimensional shape made, for example, of plastic or polystyrene and preferably light-coloured is further provided, positioned relative to the image projector to receive the projection of the image on a surface thereof.
  • a camera which may be digital, video, or film-based or the like, is further provided focused on the projection of the image on the shape surface, and arranged to capture images thereof.
  • the camera is preferably provided adjacent to but not exactly co-located with the projector, such that the camera optical axis is at a slight angle (of preferably no more than ten degrees, and preferably no more than 1 degree) to the optical axis of the projector.
  • a slight angle of preferably no more than ten degrees, and preferably no more than 1 degree
  • the projector acts to project an image, being preferably a viseme or the like, onto the surface of the shape, thereby effectively "texturing" the surface.
  • the camera being focused on the surface of the shape, captures an image of the projected image, which by virtue of the optical axis of the camera being at an angle to the optical axis of the projector exhibits the "aspect ratio change" effect provided by the invention.
  • the image captured by the camera is then output as the processed image.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Processing Or Creating Images (AREA)
  • Image Processing (AREA)
  • Image Generation (AREA)
EP02253679A 2002-05-24 2002-05-24 Procédé et système de traitement d'images Withdrawn EP1365359A1 (fr)

Priority Applications (10)

Application Number Priority Date Filing Date Title
EP02253679A EP1365359A1 (fr) 2002-05-24 2002-05-24 Procédé et système de traitement d'images
CNA2008101489274A CN101447093A (zh) 2002-05-24 2003-05-08 图像处理方法与系统
PCT/GB2003/001961 WO2003100728A2 (fr) 2002-05-24 2003-05-08 Procede et systeme de traitement d'image
JP2004508100A JP4354907B2 (ja) 2002-05-24 2003-05-08 画像処理方法及び装置
CA002484491A CA2484491A1 (fr) 2002-05-24 2003-05-08 Procede et systeme de traitement d'image
CNA038112876A CN1653490A (zh) 2002-05-24 2003-05-08 图像处理方法与系统
EP03725373A EP1508126A2 (fr) 2002-05-24 2003-05-08 Procede et systeme de traitement d'image
KR1020047018978A KR100826443B1 (ko) 2002-05-24 2003-05-08 이미지 처리 방법 및 시스템
AU2003227910A AU2003227910A1 (en) 2002-05-24 2003-05-08 Image processing method and system
US10/513,032 US7184049B2 (en) 2002-05-24 2003-05-08 Image processing method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP02253679A EP1365359A1 (fr) 2002-05-24 2002-05-24 Procédé et système de traitement d'images

Publications (1)

Publication Number Publication Date
EP1365359A1 true EP1365359A1 (fr) 2003-11-26

Family

ID=29286224

Family Applications (2)

Application Number Title Priority Date Filing Date
EP02253679A Withdrawn EP1365359A1 (fr) 2002-05-24 2002-05-24 Procédé et système de traitement d'images
EP03725373A Withdrawn EP1508126A2 (fr) 2002-05-24 2003-05-08 Procede et systeme de traitement d'image

Family Applications After (1)

Application Number Title Priority Date Filing Date
EP03725373A Withdrawn EP1508126A2 (fr) 2002-05-24 2003-05-08 Procede et systeme de traitement d'image

Country Status (8)

Country Link
US (1) US7184049B2 (fr)
EP (2) EP1365359A1 (fr)
JP (1) JP4354907B2 (fr)
KR (1) KR100826443B1 (fr)
CN (2) CN101447093A (fr)
AU (1) AU2003227910A1 (fr)
CA (1) CA2484491A1 (fr)
WO (1) WO2003100728A2 (fr)

Families Citing this family (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7543735B2 (en) 2002-01-17 2009-06-09 At&T Intellectual Property I, Lp System and method for processing package delivery
KR100657065B1 (ko) * 2004-01-29 2006-12-13 삼성전자주식회사 휴대단말기의 캐릭터 처리 장치 및 방법
JP2007534278A (ja) * 2004-04-20 2007-11-22 ボイス シグナル テクノロジーズ インコーポレイテッド ショートメッセージサービスを通じる音声
JP2006004158A (ja) * 2004-06-17 2006-01-05 Olympus Corp 画像処理プログラム、画像処理方法、画像処理装置及び記録媒体
US20070065000A1 (en) * 2005-09-21 2007-03-22 Joonpyo Hong Apparatus and method for producing character sculpture
FR2907246B1 (fr) * 2006-10-12 2008-12-12 Airbus France Sas Procede et dispositifs de projection de motifs bidimensionels sur des surfaces complexes d'objets tridimensionnels
GB2463107A (en) * 2008-09-05 2010-03-10 Skype Ltd A remote control unit of a media device for placing/receiving calls, comprising activating one of the two wireless transceivers when needed.
GB2463105A (en) * 2008-09-05 2010-03-10 Skype Ltd Viewer activity dependent video telephone call ringing
GB2463110B (en) * 2008-09-05 2013-01-16 Skype Communication system and method
GB2463124B (en) 2008-09-05 2012-06-20 Skype Ltd A peripheral device for communication over a communications sytem
GB2463109B (en) * 2008-09-05 2013-03-13 Skype Communication system and method
GB2463103A (en) * 2008-09-05 2010-03-10 Skype Ltd Video telephone call using a television receiver
GB2463108B (en) * 2008-09-05 2012-08-29 Skype Communication system and method
GB2463104A (en) 2008-09-05 2010-03-10 Skype Ltd Thumbnail selection of telephone contact using zooming
US8610761B2 (en) 2009-11-09 2013-12-17 Prohectionworks, Inc. Systems and methods for optically projecting three-dimensional text, images and/or symbols onto three-dimensional objects
US8432401B2 (en) * 2010-05-10 2013-04-30 Kabushiki Kaisha Square Enix Image processing apparatus, image processing method, and image processing program
US20150070138A1 (en) * 2012-07-06 2015-03-12 Alan Haddy Detection of buried assets using current location and known buffer zones
CN103578127B (zh) * 2013-11-13 2016-08-31 北京像素软件科技股份有限公司 一种对象转身操作实现方法及装置
JP6016765B2 (ja) * 2013-12-13 2016-10-26 株式会社ジオ技術研究所 ３次元地図表示システムおよび合成テクスチャ生成装置
CN106445551A (zh) * 2016-10-12 2017-02-22 大连文森特软件科技有限公司 可视化编程的vr星空制作与观测系统
EP3724858A4 (fr) 2017-12-14 2021-01-13 Samsung Electronics Co., Ltd. Procédé et appareil de gestion de données immersives
US10910001B2 (en) * 2017-12-25 2021-02-02 Casio Computer Co., Ltd. Voice recognition device, robot, voice recognition method, and storage medium
KR102008772B1 (ko) * 2018-01-15 2019-08-09 주식회사 스트리스 특징 기하 기반의 멀티센서 캘리브레이션 및 융합을 위한 장치 및 방법
US10573060B1 (en) * 2018-06-14 2020-02-25 Kilburn Live, Llc Controller binding in virtual domes
US10740957B1 (en) * 2018-06-14 2020-08-11 Kilburn Live, Llc Dynamic split screen
US10699705B2 (en) * 2018-06-22 2020-06-30 Adobe Inc. Using machine-learning models to determine movements of a mouth corresponding to live speech
GB201814121D0 (en) * 2018-08-30 2018-10-17 Liopa Ltd Liopa
US11030291B2 (en) * 2018-09-14 2021-06-08 Comcast Cable Communications, Llc Methods and systems for user authentication

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5933151A (en) * 1997-03-26 1999-08-03 Lucent Technologies Inc. Simulated natural movement of a computer-generated synthesized talking head
EP0992933A2 (fr) * 1998-10-09 2000-04-12 Mitsubishi Denki Kabushiki Kaisha Méthode pour générer directement des animations faciales réalistes directement à partir de la parole en utilisant des chaínes de Markov cachées
JP2001109907A (ja) * 1999-10-04 2001-04-20 Sharp Corp ３次元モデル生成装置および３次元モデル生成方法ならびに３次元モデル生成プログラムを記録した記録媒体

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5500671A (en) * 1994-10-25 1996-03-19 At&T Corp. Video conference system and method of providing parallax correction and a sense of presence
US5880788A (en) * 1996-03-25 1999-03-09 Interval Research Corporation Automated synchronization of video image sequences to new soundtracks
US5824927A (en) 1996-05-24 1998-10-20 Tonon; Thomas Keyed free-reed instruments scope
US6044168A (en) * 1996-11-25 2000-03-28 Texas Instruments Incorporated Model based faced coding and decoding using feature detection and eigenface coding
US6072496A (en) * 1998-06-08 2000-06-06 Microsoft Corporation Method and system for capturing and representing 3D geometry, color and shading of facial expressions and other animated objects
US6806898B1 (en) * 2000-03-20 2004-10-19 Microsoft Corp. System and method for automatically adjusting gaze and head orientation for video conferencing
US6661418B1 (en) * 2001-01-22 2003-12-09 Digital Animations Limited Character animation system
US6654018B1 (en) * 2001-03-29 2003-11-25 At&T Corp. Audio-visual selection process for the synthesis of photo-realistic talking-head animations
US6985148B2 (en) * 2001-12-13 2006-01-10 Microsoft Corporation Interactive water effects using texture coordinate shifting
US6828972B2 (en) * 2002-04-24 2004-12-07 Microsoft Corp. System and method for expression mapping

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5933151A (en) * 1997-03-26 1999-08-03 Lucent Technologies Inc. Simulated natural movement of a computer-generated synthesized talking head
EP0992933A2 (fr) * 1998-10-09 2000-04-12 Mitsubishi Denki Kabushiki Kaisha Méthode pour générer directement des animations faciales réalistes directement à partir de la parole en utilisant des chaínes de Markov cachées
JP2001109907A (ja) * 1999-10-04 2001-04-20 Sharp Corp ３次元モデル生成装置および３次元モデル生成方法ならびに３次元モデル生成プログラムを記録した記録媒体

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
GANG XU ET AL: "THREE-DIMENSIONAL FACE MODELING FOR VIRTUAL SPACE TELECONFERENCING SYSTEMS", TRANSACTIONS OF THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS OF JAPAN, INST. OF ELECTRONICS & COMMUNIC. ENGINEERS OF JAPAN. TOKYO, JP, vol. E73, no. 10, 1 October 1990 (1990-10-01), pages 1753 - 1761, XP000176466 *
MORISHIMA S ET AL: "A MEDIA CONVERSION FROM SPEECH TO FACIAL IMAGE FOR INTELLIGENT MAN-MACHINE INTERFACE", IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, IEEE INC. NEW YORK, US, vol. 9, no. 4, May 1991 (1991-05-01), pages 594 - 600, XP000874621, ISSN: 0733-8716 *
YUONG-WEI L ET AL: "A three-dimensional muscle-based facial expression synthesizer for model-based image coding", SIGNAL PROCESSING. IMAGE COMMUNICATION, ELSEVIER SCIENCE PUBLISHERS, AMSTERDAM, NL, vol. 8, no. 4, 1 May 1996 (1996-05-01), pages 353 - 363, XP004069966, ISSN: 0923-5965 *

Also Published As

Publication number Publication date
WO2003100728A3 (fr) 2004-07-29
CA2484491A1 (fr) 2003-12-04
JP4354907B2 (ja) 2009-10-28
AU2003227910A1 (en) 2003-12-12
JP2005527053A (ja) 2005-09-08
KR100826443B1 (ko) 2008-04-29
WO2003100728A8 (fr) 2004-10-14
EP1508126A2 (fr) 2005-02-23
KR20050033550A (ko) 2005-04-12
CN1653490A (zh) 2005-08-10
US20050162432A1 (en) 2005-07-28
CN101447093A (zh) 2009-06-03
US7184049B2 (en) 2007-02-27
WO2003100728A2 (fr) 2003-12-04

Similar Documents

Publication Publication Date Title
US7184049B2 (en) Image processing method and system
US9396588B1 (en) Virtual reality virtual theater system
US20220130111A1 (en) Few-shot synthesis of talking heads
JP4610747B2 (ja) 画像生成システム、プログラム及び情報記憶媒体
US11461942B2 (en) Generating and signaling transition between panoramic images
US20220068010A1 (en) Augmented reality methods and systems
JP4686602B2 (ja) ３ｄ画面に動映像を挿入する方法及びその記録媒体
JPWO2017094527A1 (ja) 動画生成システムおよび動画表示システム
JP2008033522A (ja) プログラム、情報記憶媒体、及び画像生成システム
CN113630646A (zh) 一种数据处理方法及装置、设备、存储介质
CN111862273A (zh) 动画处理方法、装置、电子设备及存储介质
JP2739444B2 (ja) ３次元モデルによる動き生成装置
CN116225234A (zh) 交互方法和云服务器
JP3576126B2 (ja) 画像生成システム、プログラム及び情報記憶媒体
KR100693134B1 (ko) 3차원 이미지 처리
CN115830210A (zh) 虚拟对象的渲染方法、装置、电子设备及存储介质
CN114288656A (zh) 虚拟音源物体设置方法、装置、电子设备及介质
JP4245356B2 (ja) ゲームシステム及び情報記憶媒体
JP4651204B2 (ja) 画像生成システム、プログラム及び情報記憶媒体
JP2005209217A (ja) ゲームシステム及び情報記憶媒体
JP2002035409A (ja) ゲームシステム及び情報記憶媒体
US11632538B2 (en) Object display in virtual and augmented reality
JP2007087030A (ja) プログラム、情報記憶媒体及び画像生成システム
US20230394732A1 (en) Creating images, meshes, and talking animations from mouth shape data
Casas Intermediated Reality

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

AX Request for extension of the european patent

Extension state: AL LT LV MK RO SI

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20030925