AU2020100953A4 - Automated food freshness detection using feature deep learning - Google Patents

Abstract

AUTOMATED FOOD FRESHNESS DETECTION USING FEATURE DEEP LEARNING Abstract: Accurate spoilage detection of food products such as fruit and vegetables is extremely crucial for wholesalers who store and sell massive amounts of such items. Automatic detection of fruits and vegetables via computer vision is still challenging task due to complexity, time consuming, slow process and laborious. This invention proposes a proactive research of a variety of fruit and vegetable images for automatic detection of freshness using deep learning. Deep learning is the data analysis tool to solve the problems and challenges in food domain such as recognition and quality detection of vegetables, fruits, meat, aquatic products, nutrients estimation and food recognition. The dataset consists of fruit and vegetable and fruit images that are captured by digital camera. Image resizing and enhancement can be performed during preprocessing. The novel hybrid architecture proposes YOLOV3 for object detection and Google Net inception V4 for feature learning and classification. SoftMax uses binary classes as freshness and rotten fruits and vegetables. The results show that a hybrid architecture provides high accuracy with Ft score to classify the fruits and vegetables as freshness and rotten. The ability of detection and classification of rotten fruits and vegetables will help retailers to optimize their business. AUTOMATED FOOD FRESHNESS DETECTION USING FEATURE DEEP LEARNING Precision Data Labelling TestingiData Evalnation -Recall Fl Score Image Preprocessing rFre Image Acquisition YOLOV3 Data set Creation Rotten Feature Learning+Classiication Inception V4 Fig 1: Data flow Diagram

Description

AUTOMATED FOOD FRESHNESS DETECTION USING FEATURE DEEP LEARNING

Precision

Data Labelling TestingiData Evalnation -Recall Fl Score

Image Preprocessing rFre

Image Acquisition YOLOV3 Data set Creation Rotten

Feature Learning+Classiication Inception V4

Fig 1: Data flow Diagram

AUTOMATED FOOD FRESHNESS DETECTION USING FEATURE DEEP LEARNING

Field of Invention:

The consumption of nutritionally balanced fruits and vegetables is important since they are the source of calories for all living creatures. This invention proposes the novel methodology called deep learning for automatic detection of fruits and vegetable freshness. The moldering of fruit has major effects on economic activities, and it is estimated that almost a third of the cost of fruit goes to decay. In addition, the selling of fruit may be affected because it is believed by customers that rotten fruits are harmful to human health because decreased amounts of amino acids, vitamins, sugar / glucose and other nutrients eventually increase public concerns about issues of freshness, both of which together contribute to controversies on this topic to avoid or slow down the decay process. In modern era, the classification of different foods can be done by pre-defined deep learning methods such as GoogleNet, AlexNet, and VGG Net that yielded high accuracy. This invention explains the novel hybrid model that is the combination of YOLO V3 and Inception ResNet-v2 for object detection and classifying freshness of fruits and vegetables that yields higher accuracy than conventional methods.

Background and prior art of the invention:

A traditional method was to collect multiple feature extractors and merge them to create better features. But this method requires too many optimization methodologies as well as manual labor to customize domain-specific parameters to achieve a reasonable degree of accuracy.

1| P a g e

With the dramatic development of modern society, more consideration has been developed to the quality of life, especially the food we eat. But labeling food manually is not applicable to this rapid -tempo society anymore.

Nowadays, due to economy contribution fruit freshness grading becomes more important indeed the manual operation is time consuming.

A deep vanilla neural network involves large number of parameters that is complex to train a system without overfitting the model due to the lack of a sufficient number of training samples.

Kendo explained non-dangerous automatic quality strategy to distinguish the nature of fruit without affecting the fruit. Due to poor policy, it is exceedingly difficult to differentiate the nature of the fruits based on color, shape and size.

Narvankar et al used a soft X-ray imaging on wheat grains and processing by various statistical discriminant classifiers like linear and quadratic classifiers. The features are extracted from x-ray images of the infected and healthy wheat grains and compared then density is evaluated. Indeed, x-ray technique involves high cost and care needs to be taken for shielding from radiations.

Chelladurai et al proposed thermal imaging technique used to detect fungal infections in food products. However, the proposed method could not differentiate between various fungal species as the temperature profiles of the samples that was infected by various fungal species was the same.

Ye D et al explained a hyperspectral image-based technique for detecting the blemish in potatoes. The hyperspectral image-based data was preprocessed and then optimize the modeling parameters by using grid search algorithm. The model was

21 P a g e not detecting the minor blemishes on the potatoes and necessary to extract minuscule information from data.

Detection of fruit freshness is explained by Raspberry Pi OpenCV method to detect the shape, size, and color of fruits. Edge detection algorithms are used to find the detected fruit. The main drawback of this proposed method is that only detects the outer appearance of fruits and vegetables with 80% accuracy.

Zhang et al had proposed a hybrid classification method based on fitness-scaled chaotic artificial bee colony and feed forward neural network to identify the fruits. Digital camera is used to collect the 18 types of fruits with 1653 images. The principal component analysis was used to reduce the dimension of features and stratified fivefold cross validation technique was used to enhance the FNN to achieve high accuracy. However resizing images degraded the image quality and it raises the misclassification rate.

Lu et al presented a fruit classification tool based on computer vision and artificial intelligence. The misclassification rate is reduced by single-hidden layer feed forward neural network is proposed. This methodology consists of two phases as online and offline prediction. However, two types of fruits were not identified well and this technique used only one hidden layer so the weights/biases cannot be interpreted.

Mahendran et al demonstrated the application of image analysis and computer vision system for quality evaluation of fruits and vegetables. Computer vision provides rapid, economic, and objective assessment but difficulties still exist. In all sectors, slowly updates the computer vision technology and processing speed leads to meet modem requirements.

31Page

Lu et al proposed machine learning algorithms for detecting the diseases that affects the rice plants. An image of healthy and affected rice leaves and stem was captured. The healthy and affected rice leaves were fed into CNN to train a model using different hyper parameters to obtain optimal prediction accuracy. But this technique is overly complex for detecting all the diseases with high accuracy.

Z May et al proposed a grading system is to differentiate various classes of oil palm fruit. The system coding was developed for segmentation of colors, mean color intensity and the decision making using fuzzy logic to train the data and make the classification. The image processing coding part is overly complex to implement.

Lopez Jose et al proposed automatic classification of fruits using vision systems to detect the damages in the citrus peel and classifies the type of flaws presented.

CHOi et al. had explained a framework for evaluating and grouping the fruits based on appearance and flavors. This methodology uses image processing techniques for computing appearance and near-infrared spectroscopy for estimating flavors. After that the CNN is suggested for fruit grading classification. However, the production cost is to be decreased and will improve the efficiency.

Objective of the invention:

The main objective of this invention is to detect the freshness of fruits and vegetables by deep learning.

• To create the dataset for fruits and vegetables by snapping Images with high quality cameras.

41Page

• To detect the target object, from the background, YOLO v3 network is proposed and classify the fruits and vegetables by building a deep learning based model called Inception-ResNet-v2. • To calculate the classification accuracy SoftMax is used. Recognizing the freshness of each fruit and vegetable from image with less computation and high accuracy.

Summary of the invention:

Data Collection:

The images are captured by using high quality digital camera from various sources and environments. The dataset contains 5 types of fruits and 5 types of vegetables. Totally 10000 images are collected as freshness and rotten with each type of fruits and vegetables about 500.

Data preprocessing:

Many of the images are of low quality, as blurred and low exposure to light. Once the images are collected, it will be pre-processed such as image enhancing and resizing. Image enhancement approaches were taken to improve the quality of the images. The contrast enhancement allows the revelation of latent information for too much or too little ambient light exposure. Contrast enhancement allows the transparency of residual data for too much or too little ambient light exposure.

Object Detection:

You Only Look Once known as YOLO is one of the fastest real-time object detection algorithms. The approach includes a single deep convolutional neural network that segments the input into a cell grid and directly predicts the bounding box and the

51 P a g e classification of the entity. As a result, many candidates bounding boxes are integrated into a final prediction by a post-processing step.

Fruit images are processed in such a way that the resulting images retain regions of interest. The region of interest is in the rectangular shape image. However, unlike rectangular shape of the bounding box, most of the shapes of the objects is polygons or with rounded angles or corners. This object detection gets input from data set and splits the images into fruits and vegetable images and background images. The detected images are categorized and labeled as rotten and freshness on visual inspection.

Inception V4:

In this invention, a new deep learning model called YOLO-INCEPTION which is a novel hybrid model that utilizes YOLO whose predictive bounding boxes are fed to the Inception V4 CNN for freshness grading. Dataset images fetches into YOLOv3 for object detection, where the measurements of rectangular box are determined.

The Inception architecture is highly tunable; because of changes in the number of filters in the various layers that does not affect the quality of the fully trained network. In order to achieve optimal training speed tuned the layer sizes in terms of balancing the computation between the different sub-networks of the model. Inception-v4 attempts to disperse of this undue baggage and makes consistent choices for Inception blocks for each grid size.

All the non-V-marked convolutions in the figures are the same-padded indicating that their output grid matches the size of their input. Convolutions marked with "V" are valid padded, that input patch of each unit is fully contained in the previous layer and the grid size of the output activation map is reduced. It can be trained without partitioning the replicas, with memory optimization to backpropagation. 61Page

The input of inception v4 network is an image (299 x 299pixels), the output is two classes one for freshness and other for rotten. The final layer called Average Pooling and its output is a1-Dimension of 1,536 floating numbers then three logit features are added to final features so the overall output is 1,539 floating numbers for each image. SoftMax layer is used to classify the fruits and vegetables image as a freshness or rotten. During the test phase the 30% images are used for evaluation purpose. Accuracy and F1 score will be calculated. This invention will produce best accuracy, F1 score and less error rate than the conventional deep learning methods.

Statement of the invention:

The main aim of this proposal is for automatic freshness detection of fruits and vegetables using deep learning. YOLO V3 is used to detect the fruits and vegetables from the image dataset and split it into fruits and vegetables and background images .A Deep learning based Convolutional neural network model called Inception v4 will be trained on the images, collected from a digital camera, preprocessed the obtained image and labelled. The performance of the Deep learning model will be evaluated based on F1 score and classification accuracy.

Brief description of the Drawings:

Fig 1: Data flow Diagram

Fig 2: YOLO V3 architecture

Fig 3: Inception-V4 Architecture

71Page

Detailed description of the drawing:

In figure 1 presents the overall block diagram for automatic detection of fruits and vegetables freshness using deep learning. Image acquisition consists of two steps such as capturing image and preprocessing. The images are captured by using digital camera. The captured image consists of lot of noises such as blurred image, low contrast image. Totally 10000 images of fruits and vegetables are captured with 5 different types each of 500. Dataset contains both freshness and rotten images of fruits and vegetables.

Data preprocessing is used for removing noise, enhance the images, enhancing contrast, light effect and resizing the captured images and stored in the dataset. Further YOLOV3 is used to detect the images of fruits and vegetables and separates from the background image. It is best object detection methodology consists of single feed forward network to classify the detected objects and labelled based on visual inspection as freshness and rotten. During training phase the inception v4 is proposed for feature learning and classification. Finally softmax layer uses binary class to classify the given image from the dataset as freshness and rotten. The performance metrics of classification network like accuracy and F1 score will be calculated and provides better accuracy values than traditional methods.

The figure 2 explains the YOLO V3 used for object detection and extracts the fruits and vegetables images from the background. The approach includes a single deep convolutional neural network that segments the input into a cell grid and directly predicts the bounding box and the classification of the entity. As a result, a large number of candidate bounding boxes are integrated into a final prediction by a post processing step.

81Page

The figure 3 explains the architecture of Google Net Inception V4. Inception V4 consists of two parts, feature extractor and full-connected layer. In general terms, the feature extractor has many coevolutionary blocks, including one Stem block, four Inception-A blocks, seven Inception-B blocks, three Inception-C blocks and one Average Pooling layer. Integration of drop-out block and SoftMax layer is known as fully connected layer.

The stem module uses Convolution and MaxPool block for converting 299 x 299 x3 image shape into 35 x35 x384 image shape which is the input of Inception-A block. Inception-A, Inception-B, Inception-C blocks use only Convolution and Average Pooling to convolute higher abstract features of images. While Inceptions with same type and same shape size are connected directly in sequence, Inceptions with different type need a reduction grid-module to connect. Consider example, for connecting Inception-A block to Inception-B block, Reduction-A exploits grid reduction module which transforms a 35 x35 shape to a 17 x 17 shape. Similarly, Reduction-B grid-reduction module which converts a 17 x17 shape to an 8 x 8 shape is used to connect Inception-B block and Inception-C block. Average pooling layer converts the output (8 x 8 x1536 shape) of Inception-C block into 1 Dimension of 1536 features.

In Inception-V4 schema, indeed the features of average pooling layer is changed by dropout layer (keep 0.8) then trained or classified by SoftMax as full-connected layer. The final output is binary classes for freshness and rotten of fruits and vegetables. The features are saved to a file as output of feature extractor component.

91Page

Claims (1)

1. Editorial Note 2020100953 There is only one page of the claim

   AUTOMATED FOOD FRESHNESS DETECTION USING FEATURE DEEP LEARNING

   We claim that,

   • An electronic device with touch screen such as smart phone, laptop, tablet etc. with high speed internet or Wi-Fi. • A high-quality digital camera is used to capture the fruit and vegetable images. • Dataset consists of 5 types of fruits (Strawberry, Banana, Apple, Pear and Mango) and 5 types of vegetables (Tomato, Potato, Chili, Carrot and Brinjal). Dataset contains 10000 images in which 500 fresh and 500 rotten images for each type. • Preprocessing techniques such as resizing, enhancing are applied to the obtained images of fruits and vegetables. • YOLOV3 is used for detecting the fruits and vegetables from images and splitting into fruits and vegetables and background images. • Google Net inception V4 which has high accuracy and low computation, extracts deep features of fruits and vegetables, and classifies as fresh and rotten using SoftMax layer.

   Editorial Note 2020100953 There is only four pages of the drawing

   AUTOMATED FOOD FRESHNESS DETECTION USING FEATURE Jun 2020

   DEEP LEARNING 2020100953

   Fig 1: Data flow Diagram

   AUTOMATED FOOD FRESHNESS DETECTION USING FEATURE Jun 2020

   DEEP LEARNING 2020100953

   Fig 2: YOLO V3 Architecture for object detection

   1|Page

   AUTOMATED FOOD FRESHNESS DETECTION USING FEATURE Jun 2020

   DEEP LEARNING

   2|Page Jun 2020 2020100953

   Fig 3: Inception V4 architecture