GB2581167A - Video processing apparatus - Google Patents

Abstract

A method and apparatus for obtaining and processing a video recording or sequence of images of a human subject 12 to determine measurements of that subject comprises obtaining a recording of the subject, manually providing a measurement of the body of the person, using that measurement to scale a point cloud representing the subject which is obtained from the video recording, through identification of the subject and key parts (for example the head, torso, and/or limbs) of the body of the subject and placing markers on those body parts, to measure body parts of the subject. These measurements may be used in the fitting of clothes to that subject. The method may comprise identifying one or more front, back, and/or side view of the subject using the width of the subject in the image as a guide. The method may also comprise placing markers on the boundaries and/or extremities of the identified body parts, which may be identified by referring to a set of identification rules. The method may be implemented using an application (“app”) installed on a device with a camera 22 such as a smartphone 20.

Description

Video processing apparatus

FIELD

Embodiments disclosed herein relate to the computer implemented construction of a data structure representing a three dimensional model of a human user from a video recording capturing a sequence of images of the user in a suitable pose. From this model, measurements of the subject human user can be generated, for future use.

BACKGROUND

Historically, consumers obtained items of clothing from a tailor or couturier. Such clothing would be fitted to the customer, either on a purely bespoke basis or at least involving personalised fitting to the person. To do this, the maker would take accurate measurements of the person, to enable an article of clothing to be made, or fitted, to the person's individual requirements.

This approach proved unattainably expensive for most consumers. Thus, in order to satisfy more impecunious sections of the market, a clothing retail sector developed, wherein retailers would sell clothing on a "ready-to-wear" basis. Articles of clothing are offered for sale in a variety of predetermined sizes. These sizes are determined to meet the overall demand of the market in question.

However, sizing is not standardised from one retail outlet to another. Even within a single retailer's range, different items may be sized differently. It can be difficult for consumers to identify articles of clothing that actually fit them, from the stated size alone. It can be necessary to try on large numbers of articles of clothing before purchase, and this can be time-consuming and frustrating. The fit of clothing can be subjective, and different items, which may ostensibly be made to fit the same model, may in fact have wholly different aesthetic impact depending on how the fabric of the clothing falls about the body of the individual concerned.

This problem is compounded by the onset of internet-based or enabled commerce. In these circumstances, the opportunity for consumers to try clothes on, before purchase, is eliminated, or at least made extremely complicated and difficult. It might involve delivering several items of clothing to a consumer, only for that consumer to reject most if not all of the items and then to have to make arrangements for their return to the supplier. This involves significant inconvenience, administration and redelivery costs.

It also introduces significant risk of fraud or diversion of goods, which would be to the commercial disadvantage of retailers and may have a knock-on effect on pricing of goods to consumers.

Further, internet-based commerce can be perceived as a commercial threat to tailoring and couture. Increasingly, commercial pressure makes it difficult for such trades to continue to operate in their traditional way. Whereas, in the past, tailors or couturiers would have trade premises at which customers could attend for measurement and fitting, increasingly it is becoming customary for such traders to make visits to customers' premises. This can be time-consuming and costly. The opportunity to arrange personal measurement services to customers is practically eliminated by internet commerce. The reader will also appreciate that internet-based commerce also presents significant commercial threat to retailers operating from physical customer-accessible premises, again due to substantially lower overhead costs.

It is thus desirable to obtain technology which ameliorates the capture of measurements of individual bodies, to aid in the fitting of clothing thereto.

SUMMARY

Embodiments are disclosed which substantially conform to the subject matter of the claims appended hereto.

DESCRIPTION OF DRAWINGS

Figure 1 provides a schematic view of operation of an embodiment; Figure 2 is a schematic structural diagram of a smartphone of the embodiment; Figure 3 is a schematic structural diagram of a network including the smartphone of figure 2; Figure 4 is a schematic functional diagram of the smartphone of figure 2; Figure 5 is a functional diagram of a measurement converter of a server of the network of figure 3.

DESCRIPTION OF EMBODIMENTS

In general terms, an embodiment described herein provides a device and method that enables creation of a three dimensional articulated avatar of the user.

In particular, an embodiment described herein comprises the extraction of a person's measurements (the measurement derivation system) from a portion of video footage recorded of that person. Such video footage may be recorded on a simple video recorder, such as implemented on a smartphone. Video recording technology may be implemented so that it can be initiated by the subject of the video, or by an assistant.

The video footage can be processed into a measurement chart of sufficient precision for the purpose of fitting clothing on the person. The processing can enable creation of an avatar in that person's own image.

Figure 1 depicts a typical arrangement 10 for capture of a suitable video recording of a subject user. A user 12 is shown in silhouette, within the field of view of a camera 22 of a smartphone 20. The smartphone 20 is shown positioned at a suitable height to enable a balanced view of the subject to be captured. In this example, the smartphone 20 is placed on a table 14, though other arrangements will be apparent to the reader.

The architecture of the smartphone 20 is shown in figure 2. As shown, the smartphone 20 comprises the aforementioned camera 22, and also a variety of other electronic components. Only those components material to this disclosure are illustrated. Thus, the smartphone 20 comprises a processor 24, a memory 26, a storage device 28 (which, as in many cases, can include a detachable storage device such as a storage card), a graphic display driver 30, a screen 32, audio input/output devices 34, a wireless communication driver 36 and an antenna 38. These various functional components are interconnected by means of a bus 40.

The reader will appreciate that this arrangement is inevitably schematic, and other components, such as a power supply and means for storing and regulating electrical power will also inevitably need to be provided but are omitted from Figure 1 for reasons of clarity.

The screen 32 is touch-sensitive, allowing detection of user input actions in cooperation with screen displays generated by the graphic display driver 30 on the screen 32 under the instruction of installed software.

In use, as will be understood by the reader, the storage device 28 stores programs and files, for use in operation of the smartphone. The processor 24 accesses the storage device 28 and transfers long-term data, such as programs and files, to the memory 26 for short-term access, i.e. moment-to-moment operation. The processor 24 accesses data from the memory 26 in execution of programs, to edit files or to store new data in new files, and to toggle between different programs.

One program which the smartphone 20 will be executing substantially at all times during operation is an operating system 50. The operating system provides a platform comprising a number of interfacing facilities of which use can be made by devisers of other programs. This enables a smartphone to operate several programs at the same time, and also to enable programmers to develop programs without recourse to information as to the hardware capabilities or configuration of the smartphone itself.

These programs are represented in figure 2 collectively as "user applications" 52.

The smartphone 20 is placed in a network with a server 80, as shown in figure 3. The network may be established by any conventional means, such as enabled by wireless communication. The server 80 is implemented by means of general purpose computing facilities. The server 80 hosts a measurement converter 82 which is a program which is capable of converting a video footage file of suitable format into a set of measurements of the filmed subject user, stored in a measurement file 84. This can then be transmitted back to the user's smartphone 20. The measurement file 84 could also be transmitted to third parties, such as web-based retailers, so that suitable configuration can be made of a clothing selection or measurement process. This could be useful to ease the shopping experience for the user.

As shown in figure 4, the smartphone 20 can be depicted in functional terms, with reference to software stored in memory 26 and storage device 28, executed by the processor 24.

The smartphone 20 comprises a video camera driver 60 operable to capture and process a sequence of still images which, in combination, form a video stream. In this embodiment, the video camera driver 60 is operable to capture a video stream of the user performing a 360 degree turn within the viewing range of the video camera.

The processor 24 is operable to execute a program, in this context also known as an app 52 (a commonly accepted abbreviation for "application"), which serves a user interface to the screen, presenting information enabling user input action. The app 52 has two main functions, namely to provide functional on-screen selectable display elements inviting user input action, and to facilitate communication of captured video information to a server environment, to be described in due course. The aforementioned communications driver 36 provides on-device communications facilities, for use by the app 52 and any other apps hosted on the smartphone, for communication with other devices such as on a wireless communications network.

Operation of the app on the smartphone will now be described. The user, using the smartphone's usual operating system user interface, activates execution of the app 42. This may be by the user digitally selecting an on-screen symbol corresponding to the app -generally, smartphone user interfaces display a plurality of such symbols, each representative of an app installed and available for use on the smartphone.

Selection of the on-screen symbol for the app 52 thus causes the smartphone 20 to execute the app 52, which causes generation of an app user interface on the display. This can be a full screen representation, or it could be a window occupying only a portion of the available display space.

The app user interface comprises a first screen which provides on-screen instructions to the user, explaining how the smartphone 20 should be deployed so that its video camera is placed at an appropriate height (such as on a table) and substantially in a vertical position. This will ensure that the principle line of sight of the video camera will be substantially horizontal, at a height of roughly a metre above floor surface, and this broadly ensures that the camera is positioned so that it can capture a full height image of a user, and that the captured image does not suffer foreshortening or other forms of distortion.

To ease capture of the body shape of the user, the user is instructed to dress appropriately. This will normally comprise skin-tight undergarments, not so tight as to cause undue compression of the torso. This will mean that an accurate representation of the natural body shape and profile of the user can be obtained.

The app user interface comprises an on-screen button, for selection by the user, to start the video capture process. The user then stands an appropriate distance from the camera, and rotates on the spot through 360 degrees while the camera captures a video thereof. This distance may be determined by trial and error, or may be suggested in the instructions to the user.

Once the video recording has been captured, the user is prompted to input his/her height. This height measurement provides a reference from which all other body measurements can be derived through analysis of the captured video stream.

It will be appreciated that any type of digital camera recording device such as a webcam, laptop, tablet or smart phone can be used to capture a video recording of the user's body.

It is envisaged that the presently described embodiment is capable of deriving a set of 20 measurements of the body. However, the disclosure is not limited to that criterion, and more or fewer measurements may be derived, depending on the implementation.

Once the video footage of the user rotating through the requisite full turn has been captured, the footage is placed in a data file which is transmitted to the server 80 which executes the measurement converter 82, to process the footage to produce a point cloud establishing an estimate of the shape of the user, along with accurate real-world measurements for garment fitting. The measurements are stored in a measurement file 84, for return transmission back to the smartphone 20 and any other indicated destinations.

The operation of the measurement converter 82 will now be described with reference to figure 5 Firstly, on receipt of the footage file, a pre-processor 90 pre-processes the footage. This stage uses standard computer vision techniques to prepare the footage for further processing. In particular, in this described embodiment, the pre-processor 90 is operable to separate the footage into individual frames and apply image corrections.

Video frames are extracted and exported to individual images. In an embodiment, images are stored in bitmap form. Resolution and dimensions are saved. Footage and frame information are saved as records in a database file for processing.

Image corrections are applied. In particular, compensation is applied to account for lens distortion. This has the potential to improve measurement accuracy. In some implementations, footage data will include metadata describing the hardware used to capture the footage. This may include information on the lens employed. When lens information can be extracted from footage metadata, compensation algorithms are adapted to the specific lens, increasing accuracy.

When lens information cannot be retrieved, transformations will be based on calculated average properties of camera equipment used in a range of devices habitually used for personal purposes. This average could be obtained by review of popular devices, such as smartphones. For instance, it might be found that a typical camera will have a functional equivalent to a 28mm focal length, and so this could form the basis for any transformation that is applied.

This process decreases the error of the final derived measurements by approx. 0.5 percentage points. That is, if the error would have been 1% without these corrections, it becomes 0.5% Each frame is then processed to separate the subject from the background, in a subject/background identifier stage 92. This is achieved as follows.

Using computer vision algorithms on frames, pixels are identified as belonging to either the subject (Subject Pixels) or the background (Background Pixels). This can be achieved by comparing frames to determine parts of each frame which are effectively invariant, in comparison with parts of each frame which vary over time. The former can be determined as background, the latter as subject.

The subject's input height in mm is processed with respect to the subject's height in the frame in px, producing the pdmm ratio to be used for later calculations.

The px/mm ratio along with a map of the subject's contours (contour map) is saved in the frame record, described with vectors. This is thus known as a contour vector map for the subject.

Using the contour vector maps, the view angle on the subject is calculated for each frame, in an angular motion marker stage 94.

Start frame of spinning motion is determined by applying motion-and shape-change detection between the frames' contour maps.

The user will be expected to face the camera at the start of the recording process. However, the reader will appreciate that the user's perception of his/her facing the camera will be subject to human error and, as will be explained in due course, this can be resolved by computer processing.

The start frame is marked as 0 degrees of rotation. A 90-degree rotation frame is found by determining where the contour map has the narrowest horizontal displacement. A 180-degree rotation frame is found by determining where the contour map's horizontal displacement peaks after the 90-degree rotation frame. A 270-degree rotation frame is found by determining where the contour map's horizontal displacement reaches minimum after the 180-degree rotation frame.

A 360 rotation frame, and the end frame for rotation, is found by determining where the contour map's horizontal displacement peaks after the 270-rotation frame.

Should it be found, after processing the whole rotation, that a revision should be made to the nomination of the start frame as representing 0 degrees of rotation, this revision can then be made at this stage.

Each block of 90 degrees of rotation is processed as follows.

The relative rate-of-rotation is found between each frame, using motion and shape-change detection algorithms.

The rotational value from 0-90 degrees in the given block is determined by subdividing the block by number of frames, times the rate-of-rotation value found above Each frame record is updated with the frame's rotational value in degrees.

The processor then proceeds to commence a body part identifier stage 96. This uses the contour map vectors along with computer vision algorithms on the frames to determine the placements of body parts. This information is used to refine measurement results. Body parts are saved as objects defined with vector information.

At its most basic, the body part identifier stage 96 identifies the following classes: * Head * Torso * Left and Right arm * Left and Right Leg Body parts are identified in such a manner, that every single Subject Pixel will belong to one of the above classes.

This is performed on each frame, starting with the 0-degree rotation frame, as well as between the frames with motion tracking. The pattern-matching procedure is in broad strokes as will now be described.

The head is known to start at the top of the Subject Pixels and to extend down to the easily identifiable shape difference between the neck and shoulders. Computer implementable instructions are provided to enable recognition of this boundary.

The torso will continue from where the head ends with the subtraction of appendages.

Arms are identified from the fingertips and upwards towards the shoulders, based on expected location in the frame. Where the arms touch the torso, there are no Background Pixels in the raw Vector Map to distinguish between these two, but with information about the arm width and shape, the armhole location is approximated. The shoulder ball is included in the arm-object.

The crotch location is identified by the tapering-in of the Background Pixels and set as starting point for the legs. Continuous Subject Pixels from this location and downward are added to the leg objects.

Each frame is pattern-matched in similar ways. Motion tracking and other calculations between the frames ensure the body parts are identified at the same physical locations throughout the video.

As in real life, it is not possible to determine to the millimetre where the leg ends, and the torso starts, as this is probably an indeterminate question. However, this is not the purpose of the Body Part Identification stage, nor is it a requirement.

Once the main body parts have been identified, the data is passed to a rotational direction stage, to determine the direction through which the subject is rotating in the footage.

Both arm-objects are isolated and size measurements On px) are taken. As the subject spins from 0 to 90 degrees, one arm's pixel measurements will become larger as this arm moves towards the camera, while the other arm's pixel measurements become smaller.

With this information, it is possible to determine whether the subject is spinning left or right. This information is saved in the Footage Record.

A mark-up tool 98 then sets markers on each frame of the footage, corresponding to positions on the user's body as detected. In an embodiment, the mark-up tool sets over a 1000 markers on each frame, and tracks movement of these markers from frame to frame across the footage.

To obtain measurements of sufficient accuracy for the purpose of fitting clothing, markers are placed on the Vector Maps. For example, to obtain a measurement of circumference of the waist of the subject, markers are placed by the placement rules of waist measurements. Two markers are set on each frame, one on each side of the waist.

The waist position is determined by several rules. Example rules can determined from the following specimen process steps.

IF the lower abdomen (lowest 1/2 of the torso) tapers inwards, the markers are placed at the narrowest point, no less than 1/3 from the bottom of the torso, ELSE IF the lower abdomen tapers outwards (for a person of more significant girth) the markers are set at the widest location ELSE the markers are set exactly 1/3 from the bottom of the torso (for a subject of less determinate shape)...

These rules are taken directly from fashion measuring standards and interpreted for flat images.

Markers for circumference measurements are not placed on fixed positions on the body and tracked between the frames, but rather set to measure the cross-section of the body at the marker's placement. In other words, the measurement obtained from placing two markers in one frame is the distance between these two points as measured in a straight line through the body.

A scale mapper 100 then determines a relationship between pixels on each frame and real-world measurements. This can be described as a px/mm ratio.

Using all the gathered information, the system calculates the person's real measurements, and a point-cloud is created, which can be represented in 3D space.

Three types of measurements are made from the markers' placements: Circumference measurements: for example waist measurements; Straight line measurements, for example lower arm length; * Multi-point measurements, for example back length, including consideration of sway.

The measurement derivation stage takes input from the other stages and calculates the subject's real-world measurements.

The measurement derivation stage receives the following input (for each frame): Pixel to real world distance (px/mm) ratio Subject rotation * Marker locations For circumference measurements, calculations are performed from frame to frame: 1) The straight-line distance between marker 1 and marker 2 of frame 1 (line 1) is calculated in pixels.

2) This distance is converted from pixel measurements to real world measurements in mm 3) The same is done for the markers in frame 2 (line 2) 4) The difference in rotation between frame 1 and frame 2 is determined, and line 2 is offset by this value from line 1.

5) Using trigonometry, the distance between marker 1 in frame 1 and marker 1 in frame 2 can be calculated in real-world measurements.

6) The process is repeated between frame 2 and 3, then 3 and 4 etc. until all frames are analysed 7) The resulting figure is an accurate circumference measurement of for example the subject's waist.

In certain circumstances, artefacts such as motion blur will be detected. Thus, it may arise that markers for a single measurement cannot be placed in a frame due to these problems. In these cases, in an embodiment, markers are not set by computer vision. Instead, algorithmic motion prediction is employed, as will be described below. In doing so, some loss of accuracy may need to be accepted. As a result, markers can be set for all frames, using different techniques if necessary. If different, less accurate techniques are employed to place markers in certain frames, the inaccuracy thus inserted into the process can be managed using a compensatory algorithm. An example of such a compensatory algorithm will now be described.

The main purpose of the compensatory algorithm of the present embodiment is to reduce the amount of error caused by 'disappearing' body parts and/or measurement tape on the body. A number of frames may be interesting for calculations except that the focused part is obscured by some rotation degree. This is where the compensatory algorithm comes in, to help remove the error that should not be taken into account. The algorithm works by determining the frames in which markers could be missing, calculating the percentage deviation error caused by these frames and compensating by applying these values to the result for more accurate determination of deviation.

Algorithmic Motion Prediction is employed for solving the 'disappearing' marker problem and complements the Compensatory Algorithm. This is also designed to be used with the normally seen marker placement by setting a second marker on the source frame from which start motion prediction is to be started. On the following and subsequent frames, a marker is set from any of the initial two markers. This new marker should represent the same position and object that was marked on the preceding source frame. When the motion prediction is executed, another marker will be predicted and marked on the current frame using the same distance value from the two source markers of the source frame. After motion prediction is completed, the current frame will contain two markers which are the manual marker and predicted marker. This current frame will now be the source frame for the next consecutive frame (if applying motion prediction). This algorithmic motion prediction feature helps reduce the high error value caused by the disappearance of body parts of interest and/or measurement marker tape. When combined with the Compensatory Algorithm it helps reduce the error value further.

In addition to producing accurate circumference measurements, the system also calculates marker positions in 3D space, thus giving accurate shape information.

This means that, not only does the system provide information about the waist's circumference, but also about the shape; whether the waist is protruding forward, or if it is sideways wide, but flat on the front. Topographic features are also registered. Such topographic features may relate to the overall body shape of the subject, or particular features of the subject. So, for example, the entire body shape may be registered, but also attention may be focused on single features such as the nose, its shape, size and how it is disposed with regard to the rest of the subject's head.

The circumference calculation gives enough information for garment fitting, while topographic information may allow for the creation of accurate 3D avatars.

It is to be appreciated that the claims appended hereto are to be interpreted in light of the foregoing description, but that no statement in the description should be held to limit the claims.

Embodiments have been described above with the aid of functional building blocks illustrating the implementation of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.

The foregoing description of the specific embodiments will so fully reveal the general nature of the invention that others can, by applying knowledge within the skill of the art, readily modify and/or adapt for various applications such specific embodiments, without undue experimentation, without departing from the general concept of the present invention. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It is to be understood that the phraseology or terminology herein is for the purpose of description and not of limitation, such that the terminology or phraseology of the present specification is to be interpreted by the skilled person in light of the teachings and guidance.

The breadth and scope of the claims should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

It will be understood that the invention is not limited to the embodiments above-described and various modifications and improvements can be made without departing from the concepts described herein. Except where mutually exclusive, any of the features may be employed separately or in combination with any other features and the disclosure extends to and includes all combinations and sub-combinations of one or more features described herein.

Claims (11)

1. CLAIMS: 1 A method of deriving a set of measurements of a human body from a first data set comprising image data forming a video clip comprising a sequence of images of said human body through a rotation about a substantially vertical axis, and a second data set comprising at least one real world measurement of the human body, the method comprising: processing the video clip so as to identify a sequence of frames, each frame being composed of a plurality of pixels; for each frame, identifying pixels as being either of the human body or of abackground;for the sequence of frames, identifying frames which correspond to the human body in frontal view, side views and rear-facing view; for the identified pixels in each frame, identifying particular pixels as corresponding to particular body parts of the human body; for the identified pixels in each frame, placing markers with respect to the identified body parts, so that correspondence can be established between pixels in a plurality of the frames for that feature; processing the markers so as to construct a data set defining a point cloud corresponding to a three dimensional representation of the human body; scaling the point cloud with respect to a scaling factor, the scaling factor being derived from analysis of at least one of said frames to determine a mapping between pixel measurements and the one or more real-world measurements; and deriving at least one measurement from the scaled point cloud.
2. 2 A method in accordance with claim 1 wherein a pixel is identified as being either of the human body or of a background by processing a plurality of said frames so as to identify a background image estimate, comparing said pixel with said background image estimate and determining whether or not said pixelcorresponds with said background image estimate.
3. 3 A method in accordance with claim 1 wherein a frontal frame, corresponding to the human body in frontal view is determined as the first frame of the sequence of frames.
4. 4 A method in accordance with claim 1 wherein a first side-on frame, corresponding to the human body in side-on view, is determined as a frame corresponding to a first minimum in the frame sequence of the width of the body in the frame.
5. A method in accordance with claim 1 wherein a rear-view frame, corresponding to the human body in rear view, is determine as a frame, after the first frame, corresponding to a first maximum in the frame sequence of the width of the body in the frame.
6. 6 A method in accordance with claim 1 wherein a second side-on frame, corresponding to the human body in side-on view, is determined as a frame corresponding to a second minimum in the frame sequence of the width of the body in the frame.
7. 7. A method in accordance with claim 1 wherein the body parts of the human body are identified by reference to a set of identification rules.
8. 8. A method in accordance with claim 1 wherein markers are placed at boundaries between identified body parts.
9. 9. A method in accordance with claim 1 wherein markers are placed at extremities of identified body parts.
10. A general purpose computer apparatus configured by computer executable instructions, configured to receive a first data set comprising image data forming a video clip comprising a sequence of images of said human body through a rotation about a substantially vertical axis, and a second data set comprising at least one real world measurement of the human body, the apparatus being operable to perform the method in accordance with any one of the preceding claims.
11. 11. A computer program product comprising computer executable instructions, to cause a general purpose computer apparatus to become configured to perform the method in accordance with any one of claims 1 to 9.