WO2021060971A1

WO2021060971A1 - A method for augmenting a plurality of face images

Info

Publication number: WO2021060971A1
Application number: PCT/MY2020/050063
Authority: WO
Inventors: Farshid PIRAHANSIAH; Hamam MOKAYED; Hock Woon Hon
Original assignee: Mimos Bhd
Current assignee: Mimos Bhd
Priority date: 2019-09-27
Filing date: 2020-07-29
Publication date: 2021-04-01
Anticipated expiration: 2022-03-27
Also published as: MY203560A

Abstract

The present invention relates to a method for increasing data for face analysis in video surveillance. The method comprises the steps of acquiring at least one face image from an image acquisition module (102), acquiring a plurality of face images available on the internet using a data input module (104), increasing face images by at least one data augmentation module (106 and 107), generating a plurality of face images based on a trained Generative Adversarial Network, GAN technique by using a GAN module, selecting proper images based on quality of the face images using a fuzzy logic module (111), saving the selected images into a fifth database, and training the deep learning module (113).

Description

A METHOD FOR AUGMENTING A PLURALITY OF FACE IMAGES

FIELD OF INVENTION

The present invention relates to a video surveillance system. More particularly, the present invention relates to a method for augmenting a plurality of face images in video surveillance.

BACKGROUND OF THE INVENTION

Video surveillance in a security system is becoming very important due to the increasing crime, violence, obscenity, and theft. Conventionally, a number of surveillance cameras are installed in enclosed spaces such as lift, bank or jewellery store to monitor the occupant. The typical surveillance cameras providing a live video stream to at least one display and capture at least an image from one angle. Therefore, a full view of face image of a person cannot be retrieved. This may result in substandard face images produced and lead to misidentification. Hence, a method for augmenting a plurality of face images is used specifically in video surveillance.

An example for augmenting a plurality of face images is disclosed in a United States Patent Publication No. 2017/0277955 A1 which relates to a system and method for acquiring a plurality of images. A video identification method is configured to identify face images monitored by a video identification device. The system acquires information of the face images to complete recognition and examination. The method includes pre-processing a plurality of face images and applies data augmentation. The method then generates face images with related transformations. Finally, the system determines the identified images using an optimised identification model in convolutional neural network.

Another example for augmenting a plurality of face images is disclosed in a Chinese Patent Publication No. CN107220600A which relates to a picture generation method based on depth learning and generative adversarial network. A generation network role is to generate human facial images using real picture distribution network. The method uses an image database with a plurality of visual images and each real picture has a unique class label. Therefore, not only the images can be generated, but also human facial images based on specific categories such as emotion and facial expression can be obtained. Although there are systems and methods developed for augmenting a plurality of face images, the need for accurately augmenting the face images still persist due to usage of well-known dataset may not be suitable for training model of the data augmentation, insufficient self-collected dataset for the training model of the data augmentation, and face attributes captured by the surveillance camera has different viewpoint to the face attributes of the well-known dataset.

SUMMARY OF INVENTION

A method for augmenting a plurality of face images is provided by the present invention. The method is characterised by the steps of acquiring at least one face image from an image acquisition module (102); acquiring a plurality of face images available on the web using a data input module (104); increasing face images by at least one data augmentation module (106 and 107); generating a plurality of face images based on a trained Generative Adversarial Network, GAN technique by using a GAN module (109); selecting proper images based on quality of the face images using a fuzzy logic module (111); saving the selected images into a fifth database (112); and training the deep learning module (113).

Preferably, the step of increasing the face images using data augmentation modules (106 and 107) includes generating new face images similar to the face images stored in the first database (103) and the second database (105) along with a light augmentation and/or a heavier augmentation.

Preferably, the GAN technique is trained by projecting down the face images (501) to an image representation with reduced dimensionality by an encoder (503); inputting a random vector to a decoder (504), wherein the random vector is a variable that has been sampled to obtain the smaller representation images from the encoder (503); generating at least one augmented face image by the decoder (504); determining whether the at least one augmented face image is real or fake by a discriminator unit (508); and repeating all of the steps until the decoder (504) is determined to be producing a real augmented face image. BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

FIG. 1 illustrates a block diagram of a system (200) for augmenting a plurality of face images according to an embodiment of the present invention.

FIG. 2 illustrates a block diagram of a Generative Adversarial Network, GAN, module (109) generating more face images according to an embodiment of the present invention.

FIG. 3 illustrates a flowchart of a method for augmenting a plurality of face images according to an embodiment of the present invention.

FIG. 4 illustrates an example of face images acquired from an image acquisition module (102).

FIG. 5 illustrates an example of augmented face images.

FIG. 6 illustrates an example of the generated face images by the GAN module (109).

DESCRIPTION OF THE PREFERRED EMBODIMENT

A preferred embodiment of the present invention will be described herein below with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the description with unnecessary detail.

Reference is made to FIG. 1 which illustrates a block diagram of a system (200) for augmenting a plurality of face images according to an embodiment of the present invention. The system (200) for augmenting a plurality of face images analyses sequence of images to detect face images in an enclosed space. The system (200) comprises an image acquisition module (102), a data input module (104), data augmentation modules (106 and 107), a Generative Adversarial Network, GAN, module (109), a fuzzy logic module (111), a deep learning module (113), and five databases (103, 105, 108, 110, and 112).

The image acquisition module (102) is connected to a camera (101). The image acquisition module (102) is configured to acquire face images as captured by the camera (101), wherein the camera (101) is installed at a monitored area for capturing video images of the monitored area such as lift, bank or jewellery store. The image acquisition module (102) is also connected to a first database (103) to store the face images acquired.

The data input module (104) is configured to acquire face images available on the internet. The data input module (104) extracts all face images from the internet by one or more predetermined keywords, wherein the keywords are defined as input that would result in the search and extraction of all possible face images from the internet such as “man”, “woman”, “face”, “man with glass” and etc. The data input module (104) is also connected to a second database (105) to store all the face images acquired from the internet. The data input module (104) checks the images acquired from the internet to ensure only appropriate face images are taken before storing the images into the second database (105), wherein the appropriate refers to a full-face image of a person.

The data augmentation modules (106 and 107) are configured to increase the number of face images by applying related image transformations which include random translations, rotation, and flips. A first data augmentation module (106) is connected to the first database (103) and a third database (108). The first data augmentation module (106) augments the face images captured by the camera (101). A second data augmentation module (107) is connected to the second database (105) and the third database (108). The second augmentation module (107) augments the face images acquired from the internet. The third database (108) is configured to store all the augmented face images, wherein all face images refer to the images acquired from the camera (101) and the internet.

The GAN module (109) is configured to generate the augmented face images based on a trained GAN technique. The GAN module (109) is connected to the third database (108) and a fourth database (110). As shown in FIG. 2, the GAN module (109) comprises an encoder (503), a decoder (504), a fake distribution unit (505), a real distribution unit (506), and a discriminator unit (507); wherein the encoder (503) and the decoder (504) are classified as a generator network while the fake distribution unit (505), the real distribution unit (506), and the discriminator unit (507) are classified as a discriminator network. Generally, the generator network is automatically adapted to produce face images, whereas the discriminator network determines whether the face images produced is authentic or fake.

The encoder (503) is configured for projecting down the augmented face image obtained from the third database (108) to an image representation with reduced dimensionality so that the images are easy to model. The encoder (503) is further connected to the decoder (504).

The decoder (504) is configured to generate at least one augmentation image. Particularly, the decoder (504) reconstructs the image representation from the encoder

(503) into a full image. The decoder (504) is connected to the encoder (503) and the fake distribution unit (505).

The fake distribution unit (505) obtains the augmented face images from the third database (108) and also the reconstructed face images from the decoder (504). The fake distribution unit (505) is connected to the third database (108), the decoder

(504) and the discriminator unit (507).

The real distribution unit (506) obtains the augmented face images from the third database (108). The real distribution unit (506) is connected to the third database (108) and the discriminator unit (507).

The discriminator unit (507) is configured to discriminate real distribution images from fake distribution images. Specifically, the discriminator unit (507) is trained by the augmented face images stored in the third database (108) to distinguish the fake images. The discriminator unit (507) compares the face images generated by the generator network with the augmented face images from the third database (108). The same process is repeated until the generator network generates real images that are verified by the discriminator unit (507). The fuzzy logic module (111) is configured to select proper images based on the quality of the augmented face images. A high-quality image is determined by sharpness of the face images. The fuzzy logic module (111) uses image quality assessment function, IQAF, to detect the sharpness of the face images. The IQAF applies Laplacian Equation to count high and low frequency of the face images. In order to detect the sharpness of the face images based on IQAF, the frequency is divided by resolution of the images and the range is between 0 and 1. The Laplacian Equation is expressed as below:

Laplace function (LF) = , where x and y are the standard Cartesian

coordinates of the face images and y is the source image.

The fuzzy logic module (111) selects proper images based on the quality of the augmented face images stored in the fourth database (110). The fuzzy logic module (111) is connected to a fifth database (112) to store the selected images that are to be used to train deep learning module (113).

The deep learning module (113) is connected to the fifth database (112). The deep learning module (113) is configured to evaluate the face images by extracting higher-level features of the face images. The deep learning module (113) is composed of several hidden layers of networks. For example, in the image evaluation, a first representational layer may extract the edges, a second layer may comprise the arrangements of edges, a third layer may comprise a nose and eyes, and a fourth layer may recognise that the image contains a face. The deep learning module (113) evaluates the face images based on an artificial neural network which is Convolutional Neural Networks, CNN.

Although the system (200) is described as having five databases, it may be apparent to a person skilled in the art to combine the storage of the respective face images to at least one database. In addition to that, the data augmentation modules (106 and 107) may also be combined to increase the face images by applying related image transformations to face images acquired from both the camera and the web.

FIG. 3 illustrates a method for augmenting a plurality of face images according to an embodiment of the present invention. Initially, the image acquisition module (102) acquires at least one face image from the camera (101) as in step 1000, wherein the camera (101) provides a live recording video of the monitored area. The recording video comprises a sequence of images which may include one or more face images. An example of the face image acquired by the image acquisition module (102) is shown in FIG. 4. The face image is then stored in the first database (103) as in step 1100.

Meanwhile, the data input module (104) acquires face images available on the internet as in step 1200. The face image is then stored in the second database (105) as in step 1300.

Thereon, the face images stored in the first database (103) and second database (105) are increased by the data augmentation modules (106 and 107) as in steps 1400 and 1500. In particular, the data augmentation modules (106 and 107) generate new face images that are similar to the face images stored in the first database (103) and the second database (105) along with a light augmentation and a heavier augmentation. For example, the face image is horizontally flipped or horizontally and vertically translated in size of 10% as for the light augmentation. The heavier augmentation is that the face image is transformed based on affine transformation as well as contrast and brightness adjustments. FIG. 5 shows examples of the augmented face images.

Next, the increased number of face images are then stored in the third database

(108) as in step 1600. Then, the GAN module (109) generates more face images based on the trained GAN technique as in step 1700. Examples of the generated face images are as shown in FIG. 6.

Prior to the GAN module (109) generating the face images, the GAN module

(109) is trained to produce the face images. The generator network produces face images while the discriminator network determines whether the face images are real or fake. Particularly, in the generator network, the encoder (503) collects the augmented face images from the third database (108) and projects the images down to an image representation with reduced dimensionality. A random vector is fed to the decoder (504) as an input, wherein the random vector is a variable that has been sampled to obtain the smaller representation images from the encoder (503). The decoder (504) then generates at least one augmented face image and send the image to the fake distribution unit (505). In the discriminator network, the discriminator unit (507) determines whether at least one augmented face image produced by the decoder (504) is real or fake. These steps are repeated until the decoder (504) produces the real images that are verified by the discriminator unit (507).

Once the GAN module (109) has generated more real face images, all of the output is stored in the fourth database (110) including the face images from the third database (108) as in step 1800. Thereon, the fuzzy logic module (111) selects proper images based on the quality of the images as in step 1900. The selection may be based on gender, age group, emotion, number of images, and degree of face alignments in order to get better generalization for training deep learning module (113). Then, the output images selected by the fuzzy logic module (111) are stored in the fifth database (112) as in step 2000. Finally, the deep learning module (113) is trained as in step 2100.

While embodiments of the invention have been illustrated and described, it is not intended that these embodiments illustrate and describe all possible forms of the invention. Rather, the words used in the specifications are words of description rather than limitation and various changes may be made without departing from the scope of the invention.

Claims

1. A method for augmenting a plurality of face images is characterised by the steps of: a) acquiring at least one face image from an image acquisition module (102); b) acquiring a plurality of face images available on the internet using a data input module (104); c) increasing the number of face images by at least one data augmentation module (106 and 107); d) generating a plurality of face images based on a trained Generative Adversarial Network, GAN technique by using a GAN module (109); e) selecting face images based on the quality using a fuzzy logic module (111); f) storing the selected images into a database (112); and g) training a deep learning module (113).

2. The method as claimed in claim 1, wherein the step of increasing the face images using data augmentation modules (106 and 107) includes generating new face images similar to the face images stored in the database (103) with a light augmentation and/or a heavier augmentation.

3. The method as claimed in claim 1, wherein the GAN technique is trained by: a) projecting down the face images to an image representation with reduced dimensionality by an encoder (503); b) inputting a random vector to a decoder (504), wherein the random vector is a variable that has been sampled to reconstruct the image representation from the encoder (503); c) generating at least one augmented face image by the decoder (504); d) determining whether the at least one augmented face image is real or fake by a discriminator unit (507); and e) repeating steps (a) to (d) until the decoder (504) is determined to produce a real augmented face image