EP0864140B1

EP0864140B1 - Method and apparatus for generating a visual record

Info

Publication number: EP0864140B1
Application number: EP96940029A
Authority: EP
Inventors: David Antony Crellin; John Robert Crellin; Robert John Dickinson
Original assignee: Individual
Current assignee: Individual
Priority date: 1995-11-30
Filing date: 1996-11-29
Publication date: 2000-05-03
Anticipated expiration: 2016-11-29
Also published as: EP0864140A1; WO1997020293A1; DE69608123D1; GB9524538D0

Abstract

A method of and an apparatus for generating a visual record that extends over a period of time. Data signals are transmitted to a control unit (1), the signals including picture signals from a camera (2) and signals from one or more types of sensing devices (4A, 4B, 4C). A record of only some of the picture signals is maintained. The degree of difference of at least one type of the transmitted data signals from data signals of the same type that have previously been recorded is assessed and only when the degree of difference assessed is greater than a preselected threshold amount is a record of the picture signals maintained.

Description

This invention relates to a method and apparatus for generating a visual record that extends over a period of time.
It is well known to generate a visual record that extends over a relatively long period of time by recording each frame of the record at a fixed time interval using a video camera. The recorded frames can be played back at a faster rate than they were recorded, enabling a viewer to see in a few seconds an event that may have taken place over a number of hours. At the same time as the picture frames are recorded values from one or more sensors may also be recorded. Such a method may be referred to as data-logging.
Such data-logging is satisfactory for events which are continuous at a substantially constant, but slow, rate of change throughout the period of data-logging. Unfortunately, however, many events to which data-logging would be relevant are not of that kind. For example, an event may involve several periods during which there is a rapid rate of change separated by other periods when there is very little or no change. In that case significant data relating to a period of rapid change may be lost because the rate of recording is slowed down; at the same time much of the available data recording space is used up by insignificant data from recording during a period when there was little or no change; another disadvantage in such a case may be that the amount of insignificant data is much greater than the amount of significant data so that the processing and analysis of the significant data is hampered.
WO 93/19467 describes a method and apparatus for detecting scene changes in video sequences. Scene changes are detected by monitoring the amount of information present in each video frame of the sequence. The amount of information of each frame is determined by assessing the amount of data required to represent the frame after the video data has been compressed, for example in accordance with the JPEG standard for spatial compression. The method utilises the fact that a change in the amount of information required to represent successive video frames once compressed corresponds to a change in the picture between those video frames.
EP 0528105 describes a picture image monitoring system for detecting a change in a picture image in respect of a monitored area. A filter circuit is used to extract a picture image characteristic, such as an edge of a picture image. A binary digitising circuit converts the extracted picture image characteristic into a binary picture image. A change in the picture image of the monitored area is detected by detecting a change of the binary picture image.
It is an object of the invention to provide an improved method and apparatus for generating a visual record that extends over a period of time.
According to the invention there is provided a method of generating a visual record that extends over a period of time, in which one or more types of data signals, including colour picture signals, are transmitted to a control unit and a record of only some of the picture signals is maintained, the method including the step of assessing the degree of difference of the transmitted picture signals from picture signals that have previously been recorded, and maintaining a record of the assessed picture signals in the event that the degree of difference assessed is greater than a preselected threshold amount whilst avoiding the maintenance of a record of the assessed picture signals in the event that the degree of difference assessed is not greater than the preselected threshold amount, wherein the step of assessing the degree of difference of the transmitted picture signals from picture signals that have previously been recorded comprises comparing the sum of the individual areas taken up by a given colour in the picture represented by the transmitted picture signals with the sum of the individual areas taken up by the same given colour in the picture represented by the previously recorded picture signals.
Because a record of picture signals is not maintained when there is little difference in the signal assessed from a previously recorded signal, it is possible to avoid wasting recording space with a multiplicity of recordings of substantially the same data. Instead, recordings can be made only when there is a significant change in the picture signals.
For most applications, it will be changes in the scene being viewed that it is important to capture and therefore, according to the present invention, a further recording is made whenever there is a significant change in the scene.
There are a variety of ways in which the degree of difference of a later scene from an earlier scene may be assessed and it is possible to choose a plurality of them when assessing the degree of difference. In accordance with the present invention the degree of difference in the sum of the individual areas taken up by a given colour in the picture is assessed in order to assess the degree of difference between the picture signals. Commonly the picture signal will provide an indication of the colour to be allotted to each pixel that may make up the picture; the number of colours that are available is not a factor critical to the present invention but as an example there may be 256 colours. The assessment of the sum of the individual areas taken up by the given colour may be carried out for a plurality of different colours, preferably for substantially all the colours that are recorded; advantageously the colours are divided into groups and the assessment is carried out for one or more different groups of colours.
The degree of difference in the brightness of the picture may be assessed in order to assess the degree of difference between the picture signals. The degree of difference in the overall brightness of the picture may be assessed but it is also possible, as well or instead, to assess the degree of difference in a preselected region of the picture.
Whilst it is within the broadest scope of the invention for the only type of data signal transmitted to the control unit to be a picture signal, it is preferred that the types of data signals transmitted to the control unit include picture signals and one or more other types of data signals and that a record of said one or more other types of data signals is maintained. For most applications the recorded data will be of much greater value if it includes not only a visual record but also a record of data from one or more other sensing devices. The other types of data signals may include signals relating to one or more of the following: air pressure, conductivity, electric current, distance, force, relative humidity, infra red, visible light, magnetic flux, oxygen, pH, position, potential difference, heart pulse rate, radioactive count rate, pressure, sound, speed, temperature, temperature difference and ultra violet.
Even when one or more other types of data signals are transmitted to the control unit, the picture signals may be the only data signals assessed to determine whether or not a record of the transmitted picture signals is maintained (although of course, if a record is maintained it could include other types of data signals, as well as the picture signals). Another possibility, is that one or more of said other types of data signals are also assessed and that the picture signals corresponding to the assessed data signals are the picture signals relating to substantially the same moment in time as the assessed data signals; in that case even though the picture signal may not have changed at all, a new recording of the picture signal is maintained because of a significant difference in one or more of said other types of data signals.
For some applications it may be desirable in certain circumstances to maintain a record of the assessed data signals, regardless of the degree of difference assessed, if a requirement other than the degree of difference assessed is met. The requirement may be that a certain time has elapsed since a record of the assessed data signals has been maintained. For example, in the case where a visual record is being made of a scene outside and data signals from a light sensor are being assessed it may be desirable to prevent a new picture record being maintained because of a slow change in brightness. By updating at regular intervals the record of the brightness signal to be used in the assessment, that can be achieved.
In certain applications it may be desirable to maintain an occasional record of the picture signals, even when there has been little or no change in them or any other assessed data signals. Thus, whilst it is an essential feature of the invention that maintenance of a record of the picture signals is sometimes avoided in the event that the degree of difference assessed is not greater than the preselected threshold amount, it is within the scope of the invention to maintain a record of the picture signals corresponding to the assessed data signals if a requirement other than the degree of difference assessed is met. The requirement may be that a certain time has elapsed since a record of the picture signals has been maintained. For example, a record may be made at hourly intervals regardless of whether there has been any change in the data signals.
Preferably the method further includes the step of recording the time to which a maintained record of a picture signal relates. That is of special advantage in the present invention because the spacing in time between successive maintained recordings of the picture signal is likely to be very variable.
The preselected threshold amount against which the degree of difference in the assessed data signals is compared is preferably an amount that can be selected by a user of the system. It will be appreciated that in different circumstances a different amount of difference in the assessed data signals will be a significant difference and justify the maintenance of a new record. As already indicated, more than one type of data signal may be assessed and more than one parameter of the data may be assessed for a difference. Each difference examined may have its own associated threshold which, regardless of the amount of any other differences, will cause a new record of the picture signals to be maintained, and/or a threshold may be set for a combination of two or more differences. For example, in a case where brightness and sound are being monitored, a new record may be maintained in the event of a change in either brightness or sound above a relatively large threshold amount or in the event of a change in both brightness and sound, each above relatively smaller threshold amounts.
Preferably the assessment of the degree of difference of at least one type of the transmitted data signals from data signals of the same type that have been previously recorded is carried out using those previously recorded data signals that relate to the most recent previous recording. For most applications comparing incoming signals with the ones most recently recorded will be the most advantageous method but it is possible to compare the incoming signals with ones recorded earlier or with a group of previous recordings.
In some applications, it may be advantageous to store temporarily all the incoming picture signals and to decide at a later time whether or not those picture signals should be maintained. That may allow time for other incoming picture signals to be received or allow time for one or more other types of data signals to be received and/or processed before a decision is taken on whether to maintain the picture signals. Thus, for example, a record may be maintained of picture signals received during a period immediately preceding a substantial change in the picture signal, even though the preceding signals may not be very different from ones received previously. Accordingly, the method may include the step of temporarily storing the picture signals and avoiding the maintenance of the temporarily stored picture signals corresponding to the assessed data signals in the event that the degree of difference assessed is not greater than the preselected threshold amount.
The record of the picture signals may be played back in various ways. In many applications, it is preferable for the picture signals to be played back with time intervals between playback of successive pictures being in proportion to the time intervals between recordings of the successive pictures. In that case of course the rate of playing back successive pictures will vary considerably during the playback but the audience will be given a sense of the time occupied by the various events viewed. Other possibilities are simply to display each recording separately as a still picture, and to play back the pictures with the same time intervals between them regardless of the time intervals between the recording of them.
In addition to displaying the picture signals during playback, it is preferable to display one or more other types of data signals that may have been recorded. Also, the time of the recording is preferably displayed if that has been recorded. The best way of displaying other types of data will vary according to the nature of the data. For example, temperature data may be displayed digitally and/or shown graphically, although if desired it could be shown by the colour in which the pictures were presented.
The present invention also provides an apparatus for generating a visual record that extends over a period of time, the apparatus including a control unit which includes a memory, and one or more types of sensing devices, including a camera, the sensing devices being arranged to transmit one or more types of data signals, including colour picture signals, to the control unit, and the control unit being arranged to assess the degree of difference of the transmitted picture signals from picture signals that have previously been recorded and to maintain a record of the assessed picture signals in the event that the degree of difference assessed is greater than a preselected threshold amount whilst avoiding the maintenance of a record of the assessed picture signals in the event that the degree of difference assessed is not greater than the preselected threshold amount, wherein the control unit is so arranged that, in use, the degree of difference of the transmitted picture signals from picture signals that have previously been recorded is assessed by at least comparing the sum of the individual areas taken up by a given colour in the picture represented by the transmitted picture signals with the sum of the individual areas taken up by the same given colour in the picture represented by the previously recorded picture signals.
The hardware of the apparatus can be completely conventional. For example, the camera may be a video camera, which may be of conventional design. The control unit may include a microprocessor and may be a conventional personal computer.
By way of example, certain embodiments of the invention will now be described with reference to the accompanying drawings, of which:

Figure 1: is a diagram of an apparatus for generating a visual record that extends over a period of time; and
Figure 2: is a flow diagram showing one example of a method of generating a visual record using the apparatus of Figure 1.

The apparatus shown in the drawing generally comprises a personal computer 1, a video camera 2 with a built in microphone, an analogue/digital converter 3 and three sensing devices 4A, 4B and 4C. The video camera 2 is connected to a PAL video input of the computer 1 and the sensing devices 4A, 4B and 4C are connected via the analogue/digital converter 3 and a common cable 5 to respective digital inputs of the computer 1. In the particular example described, sensing device 4A is a light sensor, sensing device 4B is a temperature sensor and sensing device 4C is a sound sensor. Each of the hardware units just described may be of an entirely conventional kind.
In use, the camera 2 continuously transmits a picture signal (and also a sound signal) to the computer 1 where it is converted into a digital form. In the particular example described a picture frame is stored at 240 x 180 resolution using a 256 colour palette. At the same time the sensing devices 4A, 4B and 4C transmit their signals to the computer.
In a conventional video logging system a digital recording of the picture signal would be stored in the memory of the computer at a regular, but relatively long, time interval. That mode of operation is possible with the apparatus described as is a mode in which the picture frames are recorded at as fast a rate as possible and a mode in which picture frames are recorded at, and only at, the command of a user. The apparatus is, however, also arranged to store the picture frame recording only when the picture has changed materially from a previous recording; one particular example of this mode of operation will now be described in further detail.
The particular example described uses a 256 colour palette and the 256 colours are grouped for the purposes of assessing a change in the picture into 32 groups. The latest picture frame received is analysed to determine the number of pixels in the frame of each group of colours and those numbers are compared with the corresponding numbers applicable to the last picture frame that has been stored in the memory of the computer. Only if the number of pixels of any one group have changed by more than a preselected amount (which amount may be an absolute number and/or a percentage change), then the computer will store the latest picture frame in its memory. The user is able to select the amount of change required to cause the recording of the latest picture frame.
At the same time as the latest picture frame is recorded so the digital inputs for the sensing devices 4A, 4B and 4C are recorded and also the time to which the recording relates, the time being derived from an internal clock in the computer 1.
The computer is also arranged to make a recording of the latest picture frame received if a preselected time has elapsed since a picture frame was recorded, even if there has not been a sufficient change in the picture to cause such a recording. In that case the latest picture frame is recorded for use in comparison with future frames to determine whether a new picture frame should be recorded, but it is not transferred to the memory storing the previous picture frames. In this way slow steady changes in the picture frame do not lead to the recording of new picture frames in the memory.
Figure 2 is a flow diagram showing the steps referred to above in summary form as a flow diagram.
The picture frames recorded as described above can be played back in any of three ways. Firstly, the frames can be played back at intervals that are in proportion to the time intervals between recording of the frames. For example, if there is a gap of 1 minute between recording of a first and second frame and a gap of 5 minutes between recording of the second frame and a third frame, the frames can be played back with a gap of ¹/₅ second between the first and second frames and 1 second between the second and third frames. Secondly, the frames can be played back at constant intervals regardless of the time intervals between recording of frames. In that case, in the example just given the gap between the first and second frames during playback would be the same as the gap between the second and third frames. The user can select the fixed interval between frames during playback. Thirdly, the frames can be played back one at a time at the control of the user, allowing each frame to be examined for as long as desired.
During playback, the readings from the sensing devices 4A, 4B and 4C are displayed digitally on the screen of the computer and the time at which the recording was made is also displayed. A facility is also provided to enable the readings to be displayed graphically.
In the example just described, changes in the signals from the devices 4A, 4B and 4C have no influence on whether or not a new set of recorded data is stored in the memory of the computer. It is, however, possible to make the decision as to whether or not to store a new set of recorded data dependent on an assessment of the amount of change in the signals from one or more of the devices 4A, 4B and 4C as well as, or instead of, on the amount of change in the picture signal. For example, the computer may be arranged to store a new set of recorded data in the event that the temperature, as indicated by the sensing device 4B, is more than 1°C different from the temperature last recorded. The computer can also be instructed to look at a combination of changes: for example, a modest change in the picture frame, which would not itself be sufficient to cause the computer to store a new set of recorded data, can be arranged to cause such storing of new data if at the same time there is a change in temperature recorded by the sensing device 4B of more than 0.5°C.
Changes in the sound input from the sensing device 4C, or from the microphone of the camera 2, may be analysed by techniques known per se and the amount of change in the signal analysed in a manner analogous to that already described with reference to the picture signal.
It will be understood that the sensing devices 4A, 4B and 4C are given merely by way of example. Instead or in addition to those devices, other sensors that are able to provide an electrical analogue or digital output can be employed.
The present invention can be performed satisfactorily employing a personal computer with a 486 DX2-50 microprocessor having 8Mb RAM and a 520Mb hard disc drive. With such equipment we have found it possible to store up to 1200 picture frames at 240 x 180 resolution as well as data from the sensors 4A, 4B and 4C.
The present invention can be employed in a wide variety of applications. Some of those applications will be mentioned below by way of example.
The study of animal behaviour: video logging as described above enables a study of animal behaviour to be carried out for a long period with minimal use of memory space during periods of inactivity but detailed recording during active periods.
The study of plant life: some plants flower rarely and quickly; video logging as described above can provide many picture frames of a flower opening without wasting frames on the period before flowering when the bud is dormant.
Time and motion studies in a workplace: usage of office equipment or manufacturing machinery can be studied, again without wasting memory space on recording during periods when the equipment or machinery is not in active use.
Process monitoring: it is already known to use certain sensors specifically to monitor a malfunction or breakdown in, for example, a manufacturing process; the use of video logging as described above will enable a much more complete picture of such events to be obtained right from the start of the event, even though it is unexpected.
Security applications: conventional security monitoring relies on infrequent video frame capture at regular intervals; the use of video logging as described above will enable many more video frames to be recorded during an incident but none when all is quiet, thus enabling a large collection of useful data to be acquired without recording an unmanageably large total amount.

Claims

A method of generating a visual record that extends over a period of time, in which one or more types of data signals, including colour picture signals, are transmitted to a control unit and a record of only some of the picture signals is maintained, the method including the step of assessing the degree of difference of the transmitted picture signals from picture signals that have previously been recorded, and maintaining a record of the assessed picture signals in the event that the degree of difference assessed is greater than a preselected threshold amount whilst avoiding the maintenance of a record of the assessed picture signals in the event that the degree of difference assessed is not greater than the preselected threshold amount, wherein the step of assessing the degree of difference of the transmitted picture signals from picture signals that have previously been recorded comprises comparing the sum of the individual areas taken up by a given colour in the picture represented by the transmitted picture signals with the sum of the individual areas taken up by the same given colour in the picture represented by the previously recorded picture signals.
A method according to claim 1, in which the assessment of the sum of the individual areas taken up by the given colour is carried out for a plurality of different colours.
A method according to claim 2, in which the assessment of the sum of the individual areas taken up by the given colour is carried out for substantially all the colours that are recorded.
A method according to claim 2 or 3, in which the colours are divided into groups and the assessment is carried out for one or more different groups of colour.
A method according to any preceding claim, in which the degree of difference in the brightness of the picture is assessed in order to assess the degree of difference between the picture signals.
A method according to claim 5, in which the degree of difference in the overall brightness of the picture is assessed.
A method according to claim 5 or 6, in which the degree of difference in the brightness of the picture in a preselected region of the picture is assessed.
A method according to any preceding claim, in which the types of data signals transmitted to the control unit include picture signals and one or more other types of data signals and a record of said one or more other types of data signals is maintained.
A method according to claim 8, in which the other types of data signals include signals relating to one or more of the following: air pressure, conductivity, electric current, distance, force, relative humidity, infra red, visible light, magnetic flux, oxygen, pH, position, potential difference, heart pulse rate, radioactive count rate, pressure, sound, speed, temperature, temperature difference, and ultra violet.
A method according to claim 8 or 9, in which the assessed data signals are the picture signals.
A method according to claim 8 or 9, in which the assessed data signals are one or more of said other types of data signals and the picture signals corresponding to the assessed data signals are the picture signals relating to substantially the same moment in time as the assessed data signals.
A method according to claim 11, further including the step of maintaining a record of the assessed data signals if a requirement other than the degree of difference assessed is met.
A method according to claim 12, in which the requirement is that a certain time has elapsed since a record of the assessed data signals has been maintained.
A method according to any preceding claim, further including the step of recording the time to which a maintained record of a picture signal relates.
A method according to any preceding claim, further including the step of maintaining a record of the picture signals corresponding to the assessed data signals in the event that a requirement other than the degree of difference assessed is met.
A method according to claim 15, in which the requirement is that a certain time has elapsed since a record of the picture signals has been maintained.
A method according to any preceding claim, in which the preselected threshold amount against which the degree of difference in the assessed data signals is compared is an amount that can be selected by a user of the system.
A method according to any preceding claim, in which the assessment of the degree of difference of at least one type of the transmitted data signals from data signals of the same type that have been previously recorded is carried out using those previously recorded data signals that relate to the most recent previous recording.
A method according to any preceding claim, further including the step of temporarily storing the picture signals and avoiding the maintenance of the temporarily stored picture signals corresponding to the assessed data signals in the event that the degree of difference assessed is not greater than the preselected threshold amount.
A method according to any preceding claim, further including the step of playing back the record of the picture signals.
A method according to claim 20, when dependent upon claim 15, in which the picture signals are played back with time intervals between playback of successive pictures being in proportion to the time intervals between recordings of the successive pictures.
A method according to claim 20 or 21, in which the picture signals and one or more other types of data signals are displayed during playback.
An apparatus for generating a visual record that extends over a period of time, the apparatus including a control unit which includes a memory, and one or more types of sensing devices, including a camera, the sensing devices being arranged to transmit one or more types of data signals, including colour picture signals, to the control unit, and the control unit being arranged to assess the degree of difference of the transmitted picture signals from picture signals that have previously been recorded and to maintain a record of the assessed picture signals in the event that the degree of difference assessed is greater than a preselected threshold amount whilst avoiding the maintenance of a record of the assessed picture signals in the event that the degree of difference assessed is not greater than the preselected threshold amount, wherein the control unit is so arranged that, in use, the degree of difference of the transmitted picture signals from picture signals that have previously been recorded is assessed by at least comparing the sum of the individual areas taken up by a given colour in the picture represented by the transmitted picture signals with the sum of the individual areas taken up by the same given colour in the picture represented by the previously recorded picture signals.
An apparatus according to claim 23, in which there are one or more other sensing devices apart from the camera, the other sensing devices comprising one or more of the following: a microphone; a light sensor; a temperature sensor; and a pH sensor.
An apparatus according to claim 23 or 24, in which the camera is a video camera.
An apparatus according to any one of claims 23 to 25, in which the control unit includes a microprocessor.
An apparatus according to any one of claims 23 to 26, arranged to carry out a method of generating a visual record according to any one of claims 2 to 8 or 10 to 22.