FR2758429A1 - METHOD AND INSTALLATION FOR PRODUCING A FILMED SEQUENCE OF A SPORTS ACTIVITY ENHANCED WITH REAL-TIME INSERTIONS OF VIRTUAL ELEMENTS - Google Patents

Abstract

The invention concerns a method for producing a film sequence of a sport activity characterised in that it consists in filming in real time a sequence of the evolution of the sport activity using a network of cameras each of which is equipped with a system generating data representing its position in space, its movements, its focusing and a clock signal; in acquiring in real time the instantaneous position of each person performing on the playing field and of the mobile object by analysing the filmed video images; in producing recurrently a matrix of digital data representing the position of each person and of the mobile object; in constructing at least a three-dimensional visual virtual element (C) of information and/or commentary; and in making a video image from the filmed video image and said constructed virtual element by inserting it in the filmed video image for the live or delayed broadcasting of a film sequence of the sport activity comprising said visual virtual element of information and/or commentary.

Description

 The present invention relates to the field of television sports broadcasts.

 The invention relates more particularly to a method of producing a filmed sequence of a sporting activity comprising a playing field on which at least one natural person, in particular a player or a competitor, and a moving object, in particular a ball .

 The invention is more particularly applicable to retransmissions of meetings of collective sports, in particular football, but it is in no way limited to this type of meetings and may also relate to other individual sports such as for example tennis, or athletics.

 The televised retransmissions of sports events, whether national or international, are meeting with growing success with viewers and are in this respect the subject of important commercial negotiations relating to retransmission rights and the advertising broadcasts which accompany them.

 Paradoxically, the content of the retransmissions, beyond the simple course of the sporting action in progress, has changed very little, in particular with regard to the complements of the images, ie what can be qualified in a way general comment (s).

 In fact, the classic retransmission of a football match, whether it be a live or deferred retransmission, comes down to the broadcasting of real filmed images accompanied by a sound commentary generally pronounced and recorded in direct by one or more commentators.

 Whatever the qualities of the commentators, professionals of television, the sports environment and / or journalism, their services are similar to those of a radio broadcast, that is to say that they are generally not in a technical situation to provide comments other than those resulting from their own vision, analysis and understanding of the game phase in progress, their perception of this game phase being generally twofold, that is to say that they can become aware of it instantly live, if they are present in the sports venue in which the meeting takes place, and at the same time on a screen or video monitor which reproduces to them the real filmed images which are broadcast for the benefit of viewers.

 In order to improve the commentary, and in particular to enrich its technical content relating to the analysis of the game, it has already been proposed to add to the commentators assistants who carry out statistical analyzes of certain aspects of the game and who put their results available to the director responsible for broadcasting the meeting and / or the commentators.

 These complements can be put in images, that is to say that they are the subject of the production of informative "panels" which are inserted during broadcasting, generally during idle times during which these panels are in turn commented orally.

 In addition, the complexity of the rules of the game, in particular collective sports such as football, rugby, basketball, etc., makes it necessary to enrich the television broadcasts with supplements which make it possible to illustrate the implementation work of the rules ensured by one or more referees, and this in order to improve the education of the spectators and the public with a view in particular to promoting a fair play and serene behavior of the public during the meetings.

 We also know that the training of sportspeople with a view to improving their performance and the quality of sports play, increasingly requires the use of television recordings, whether they are official meetings previous or training sequences, which are in turn commented on by those responsible for training and in particular by the coach.

 In this task, and despite his skill and his technical and tactical culture, the coach is faced with the same difficulties as the commentator of public broadcasts, that is to say that they do not have tools for visual commentary of images. real films other than panels and tables independent of the image broadcast.

 The object of the invention is to remedy these drawbacks by proposing a new method for producing a filmed sequence which makes it possible to match its distribution with a visual commentary in three dimensions and in real time.

To this end, the invention provides a method of producing a filmed sequence of a sporting activity comprising a playing field on which at least one natural person, in particular a player or a competitor, and a moving object, in particular a balloon, characterized in that it consists of
- film in real time a sequence of sports activity using a network of cameras, each of which is equipped with a system generating data representative of its position in space relative to a fixed reference point linked to the terrain , its movements, its optical focusing and a clock signal;
- acquire in real time the instantaneous position of each person moving on the ground and of the mobile object by analysis, by means of a computer unit, of the video images filmed by said network of cameras;
- cyclically produce a matrix of digital data representative of the position of each person and of the mobile object in said frame and store the successive matrices in a computer processing unit;
- construct at least one virtual visual element, in particular in three dimensions, of information and / or commentary by means of a computer unit for producing visual elements as a function of said data
- synchronize the network of cameras and computer units; and
- fabricate a video image from a filmed video image and said virtual element constructed by inserting it into the filmed video image so as to allow the broadcasting, in real time or delayed, of a filmed sequence of the sports activity comprising said virtual visual element of information and / or commentary.

According to other characteristics of the process according to the invention
- The stage of construction of the virtual element consists of
- perform an analysis and processing of said data to calculate information representative of the evolution over time of a person or of the moving object (speed-acceleration-trajectory-mobility-etc.); and
- build a virtual representation of this information intended to be inserted
- The stage of construction of the virtual element consists of
- perform an analysis and processing of said data to calculate the relative positions of people and / or the moving object relative to the terrain of evolution; and
- construct a "fixed" virtual visual object associated with said relative positions (shooting-position cone of a viewing angle) and intended to be inserted
- The stage of construction of the virtual element consists of
- perform an analysis and processing of said data to determine the evolution over time of at least one person and / or of the mobile object; and
- associate a virtual element of known structure with said person or with the mobile element to construct a virtual object to be inserted mobile during the course of the sequence
- The stage of construction of the virtual element consists of
- perform an analysis and processing of said data to calculate statistical information relating to the progress of the sporting activity and / or the evolution and movement of people and of the mobile object; and
- build a virtual representation of this information intended to be inserted
- The step of acquisition in real time of the instantaneous position of each person evolving on the ground consists in acquiring the position of these people in a reference frame with at least two dimensions corresponding to the dimensions of the terrain of evolution
- The step of acquiring the instantaneous position of the moving object in real time consists in acquiring the position of this object in a three-dimensional frame of reference, two of which correspond to the dimensions of the terrain of evolution
- That the step of real-time acquisition of the instantaneous position of the mobile object consists in filming the mobile object by means of two cameras belonging to said network of cameras;
- The method comprises an additional step consisting in manufacturing a sequence of virtual video images from virtual elements constructed so as to allow the delayed broadcasting of this sequence of virtual video images in association with the broadcasting of said filmed sequence.

The invention also provides an installation for implementing the method according to the invention, characterized in that it comprises
- a network of cameras, each of which is equipped with a system generating data representative of its position in space with respect to a fixed reference point linked to the terrain, its movements, its optical focusing and a signal d 'clock;
- a computer unit for real-time acquisition of the instantaneous position of each person moving on the ground and of the object;
- a computer unit for cyclically producing a matrix of digital data representative of the position of each person and of the mobile object in said coordinate system and storing the matrices;
- a computer unit to build at least one virtual visual element of information and / or comment based on said data
- means for synchronizing the network of cameras and the computer units; and
- Means for manufacturing a video image from a filmed video image and from said constructed virtual element.

 Other characteristics and advantages of the invention will appear on reading the description which follows for the understanding of which reference may be made to the appended drawings in which FIGS. 1 to 7 are diagrams illustrating examples of virtual elements inserted in application of the method according to the invention.

 The description which follows, given by way of nonlimiting example, relates to the televised retransmission of a football match.

 Such a meeting takes place on a playing field, in the open air or in a closed enclosure, on which play two teams of eleven players each, a field referee and two assistant referees called touch judges, i.e. twenty five physical persons in movement , and a mobile balloon which constitutes the mobile object within the meaning of the invention.

 The twenty-five natural person actors evolve globally in two dimensions, that is to say in the plane of the terrain which is defined, in a Cartesian coordinate system orthonormal by two perpendicular horizontal axes X and Y, corresponding for example to a lateral line delimiting the pitch, called touchline, and a transverse line called goal line. The evolutions reduced in height of the players can be neglected in the case of football or for example of rugby, but be important and taken into account within the framework of the method according to the invention for other sports such as basketball or volleyball.

On the contrary, the ball constantly evolves in three dimensions on the ground, that is to say that knowing its position requires knowing its coordinates [x, y, z] in a three-dimensional frame which consists of the frame
X, Y in which the positions of the players and the referees can be determined, completed by a concurrent vertical axis Z whose origin corresponds for example to the ground.

 It is thus possible to define a fixed geometric reference with respect to the terrain.

 The realization of a filmed sequence of the sporting action traditionally calls upon several video cameras which are generally established around the ground, in altitude and / or on the level of the ground, the cameras being able to be fixed or mobile and being in addition with focal variable.

 This known arrangement allows the director to choose from several images available simultaneously that which is broadcast instantly, in particular depending on the progress of the game.

 The method according to the invention proposes to use a network of video cameras which are of a particular type but which are the cameras which allow the conventional capture of the real video images used for the classic production of the filmed sequence, ie to say that it is in no way called upon a particular and additional network of cameras for the acquisition in real time of the instantaneous position of the actors on the ground and of the balloon.

 The cameras used are particular within the meaning of the invention because each of them comprises a system generating data representative of its position in the space of the ground, that is to say with respect to the previously mentioned reference point, movements of the camera, and its focus, ie the focal length of its optical system.

 Thanks to the use of such cameras called "motion control", and after an initialization step of each camera in the coordinate system, it is possible to use each image of each camera to analyze the content in order to acquire in real time, from the set of images supplied simultaneously by the network cameras, the instantaneous position x, y of each actor and of the balloon x, y, z. As a complement, each camera can be provided with telemetry means.

 Each camera can be of analog type, the images then being digitized, or be digital cameras.

 The notion of simultaneity for the purpose of image analysis naturally requires that the network cameras be synchronized, that is to say that each of them generates a clock signal.

 We are thus able to know at all times the position in the image of a camera of each actor and / or of the balloon and to associate with this position a triplet, or a doublet of coordinates x, y, z.

 Thanks to the arrangement of networked and coordinated cameras, a computer unit forming part of the installation for implementing the method according to the invention makes it possible to cyclically produce a matrix of mathematical digital data which are representative of the position of each actor and balloon.

 This method of capturing spatial coordinates is particularly advantageous because it uses the same cameras as those used for producing the real images and it does not require installing radio frequency transmitters on the actors and on the ball, transmitters which are not admitted in the regulations on the equipment of the players and which moreover impose to have on each ground a network of radio frequency receivers.

 Due to the twenty-four frames per second cycle used in video, a new matrix can be produced twenty-four times per second.

 The means for analyzing the video images of the network cameras are of known type (s) such as, for example, those sold by the company HYMATOM.

 According to the most common technique, the images are digitized and then analyzed pixel by pixel, an analysis which, in addition to determining the coordinates sought, is then used during the phase of insertion of the virtual elements.

 According to another method, and if the optical design and development parameters of the cameras used are perfectly known, a photogrammetric analysis of each image is carried out, without going through a digitization step, that is to say by implementing the principles of stereophotography. In this case, it is not necessary to have a large number of crossed cameras, one or two cameras being able to make it possible to reconstruct the three-dimensional coordinates of all the characteristic points of each video image, in particular by using the basic video signal. for retransmission called "broadcast" signal.

 To facilitate the acquisition of the position of the mobile constituted by the balloon, it can for example be called upon by two crossed cameras assigned to the pursuit of the balloon and its immediate environment, this capture being manual or automated.

 The image analysis means for determining the coordinates of the actors may for example include a real-time image analysis calculator for each player, all of the computers being managed by a centralization module which notably manages the passage of a player from one camera to another, which manages conflicts such as occlusions and collisions, automatically or with the help of a personal assistant.

 Processing by real-time calculation requires that the computer unit be synchronized with the network of cameras themselves synchronized.

 From said data matrices it is possible to construct at least one virtual element of information or comment using a computer unit for producing visual elements, which is also synchronized.

 This virtual and visual comment element, in particular in three dimensions, is intended to be inserted in the real video images filmed so as to allow the broadcasting in real time or delayed of a filmed sequence comprising the comment element inserted.

 The insertion of the virtual element for the production of "augmented" video images intended to be broadcast is carried out by known computer means comprising at least one computer operating a real-time insertion program.

 We will now give examples of the virtual elements intended to be inserted in order to constitute a visual commentary on the sporting action.

 On the basis of spatio-temporal data previously determined, it is for example possible to calculate the trajectories of the ball and the players, to construct a virtual representation of these trajectories and to insert them into the images as is shown diagrammatically in FIG. 1 on which the overall trajectory of the ball B has been shown on the ground and in the air, from one goal to another.

 The sequence of images including the insertion of the ball's trajectory is broadcast slightly delayed. It can also be embellished with three-dimensional virtual representations such as "walls" M which represent, for example, the level at which the first attacker or the last defender is located, these virtual elements being linked to the corresponding player, that is to say that their insertion in real time is "dynamic" in that at each instant of the sequence commented by the virtual elements, the latter are recalculated and inserted.

 In Figure 2, there is shown the insertion of a volume representation of a virtual "cone" of the shooting space available to a player at the time of his strike towards the goal. This representation could be supplemented with calculated data such as the distance of the player from the goal, the distance and the running time of the player before hitting, etc.

 Figures 3,4 and 5 show other angles of shots embellished with a virtual shooting cone with for example the addition of the trajectory of the ball.

 The representation illustrated in Figure 6 shows the use of a virtual wall for the illustration and commentary of the rule of offside, the wall M being "attached" to the last defender and moving with the latter.

 All of the data obtained from the real-time capture and analysis of the filmed images allows the insertion of many other elements of comments.

 We can for example insert the name of each player next to their image (see Figure 7).

 We can at any time associate with each player statistical data relating to their activity during the match (distances traveled, average or maximum speed, number of touches of balls, time of possession of the ball, etc.).

 This type of statistical analysis can be performed, and its results inserted globally, for each team.

 In the case of general images of the field taken from above, substantially vertically from the ground, it is possible to insert the tactical theoretical diagram of each team (player placement and movement modes), or to insert the actual playing diagram resulting from a prior statistical analysis of the data stored during an immediately preceding game period (game action, half-time, etc.).

 Other statistics such as the number and accuracy of shots on goal can also be the subject of virtual visual comments by insertion into images representing the goal of "impact" densities in and around the goal, embellished with the chronology. shots on goal, embellished with the chronology of shots on goal.

 We can also materialize virtually, in two or three dimensions, the space in which the ball has most often circulated.

 The method according to the invention also allows the realization of purely virtual sequences of the type of those produced in video games or the insertion of a virtual element allowing for example to simulate a hypothetical movement of a player compared to a real action. filmed and thus invent an alternative scenario of reality.

 Such a scenario also allows the explanation of complex game rules such as offside.

 The creation of virtual game sequences also makes it possible to "enter" the virtual camera on the ground and to rebroadcast a real prior action from another point of view, for example from the point of view of a player or the ball or the referee.

Claims (10)

 1. Method for producing a filmed sequence of a sporting activity comprising an evolution field on which evolves at least one physical person, in particular a player (J) or a competitor, and a mobile object, in particular a ball (B ), characterized in that it consists of  - film in real time a sequence of sports activity using a network of cameras, each of which is equipped with a system generating data representative of its position in space relative to a fixed reference point (X, Y, Z) linked to the terrain, its movements, its optical focus and a clock signal;  - acquire in real time the instantaneous position of each person (x, y) moving on the ground and of the mobile object (x, y, z) by analysis, by means of a computer unit, of the video images filmed by said network of cameras;  - cyclically produce a matrix of digital data representative of the position of each person and of the mobile object in said frame and store the successive matrices in a computer processing unit;  - construct at least one virtual visual element (M, C) of information and / or commentary by means of a computer unit for producing visual elements as a function of said data  - synchronize the network of cameras and computer units; and  - fabricate a video image from a filmed video image and said virtual element constructed by inserting it into the filmed video image so as to allow the broadcasting, in real time or delayed, of a filmed sequence of the sports activity comprising said virtual visual element of information and / or commentary.  2. Production method according to claim 1, characterized in that the step of constructing the virtual element consists in  - perform an analysis and processing of said data to calculate information representative of the evolution over time of a person or of the moving object (speed-acceleration-trajectory-mobility-etc.); and  - build a virtual representation of this information intended to be inserted.  3. Production method according to claim 1, characterized in that the step of constructing the virtual element consists in  - perform an analysis and processing of said data to calculate the relative positions of people and / or the moving object relative to the terrain of evolution; and  - construct a "fixed" virtual visual object associated with said relative positions (shooting cone-position of a viewing angle) and intended to be inserted.  4. Production method according to claim 1, characterized in that the step of construction of the virtual element consists in  - perform an analysis and processing of said data to determine the evolution over time of at least one person and / or of the mobile object; and  - associate a virtual element of known structure with said person or with the mobile element to construct a virtual object to be inserted mobile during the course of the sequence.  5. Production method according to claim 1, characterized in that the step of constructing the virtual element consists in  - perform an analysis and processing of said data to calculate statistical information relating to the progress of the sporting activity and / or the evolution and movement of people and of the mobile object; and  - build a virtual representation of this information intended to be inserted.  6. Production method according to any one of the preceding claims, characterized in that the step of acquisition in real time of the instantaneous position of each person evolving on the ground consists in acquiring the position of these people in a reference frame. at least two dimensions (X, Y) corresponding to the dimensions of the playing field.  7. Production method according to any one of the preceding claims, characterized in that the step of acquiring the instantaneous position of the moving object in real time consists in acquiring the position of this object in a three-dimensional coordinate system. (X, Y, Z), two of which correspond to the dimensions of the playing field.  8. Production method according to the preceding claim, characterized in that the step of real-time acquisition of the instantaneous position of the mobile object consists in filming the mobile object by means of two cameras belonging to said network of cameras.  9. Production method according to any one of the preceding claims, characterized in that it comprises an additional step consisting in manufacturing a sequence of virtual video images from virtual elements constructed so as to allow the delayed broadcasting of this sequence virtual video images in association with the broadcasting of said filmed sequence.  10. Installation for implementing the method according to any one of the preceding claims, characterized in that it comprises  a network of cameras, each of which is equipped with a system generating data representative of its position in space relative to a fixed reference point linked to the terrain, of its movements, of its optical focusing and of a signal d 'clock;  - a computer unit for real-time acquisition of the instantaneous position of each person moving on the ground and of the object;  - a computer unit for cyclically producing a matrix of digital data representative of the position of each person and of the mobile object in said coordinate system and storing the matrices;  - a computer unit to build at least one virtual visual element of information and / or comment based on said data  - means for synchronizing the network of cameras and the computer units; and  - Means for manufacturing a video image from a filmed video image and from said constructed virtual element.