WO2020075185A1

WO2020075185A1 - Automatic furniture and electronic equipment recommender

Info

Publication number: WO2020075185A1
Application number: PCT/IN2019/050739
Authority: WO
Inventors: Keshav TREHAN
Original assignee: Trehan Keshav
Priority date: 2018-10-07
Filing date: 2019-10-05
Publication date: 2020-04-16

Abstract

The system and method for automatic furniture and electronic equipment recommendation is disclosed. The components includes a smartphone with camera, virtual reality feature and a mixed reality feature which produces new environment and visualization where the physical and digital objects co-exist and interact in real time. The system is intending to dynamically replace a 3D image of furniture or electronic equipment in an image or video by a smart phone or device with proposed furniture or equipment from a website which is in 2D or 3D image. The system uses deep learning technique with which the user can train the object's dimensions to image clicked in panoramic view or any other view, where the user can use virtual reality feature to shrink or adjust the dimensions of matching object's image on the website to the image and finally replace and remove the objects in the processed image.

Description

AUTOMATIC FURNITURE AND ELECTRONIC EQUIPMENT RECOMMENDER

Technical field of the Invention

The present invention generally relates to image processing system, and more particularly to a method, a device, and a system for processing the captured image which is used for automatic prediction and recommendation of the furniture and electronic equipment in the captured image. Background of the Invention

In this fast developing world, the technological growth has advanced, which has paved way for the development of augmented reality. With the help of augmented reality, the components of the physical world can be transformed into digital world which changes the user's perception of the real world. The transformation of the physical world to digital world is done through the integration of immersive sensations that are perceived as natural parts of an environment. Augmented Reality (AR) is an interactive experience of a real-world environment whereby the objects that reside in the real-world are "augmented" by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory. The augmented reality alters one’s ongoing perception of a real world environment, whereas virtual reality completely replaces the user's real world environment with a simulated one. Augmented reality is related to two largely synonymous terms: mixed reality and computer-mediated reality.

People who want makeover of their homes, would have to go to furniture and electronic shops for physically viewing the furniture and electronic equipment which suits their home, may see the real furniture deposition in the house. Such system takes a lot of time of the users and the users might not be able to visit all the shops to select the furniture of their choice.

To eliminate the drawbacks of this system, the use of e-commerce websites is preferred which helps the users to view and buy the furniture and electronic equipment of their choice. With the help of the e-commerce websites, the users can view a wide range of products through various websites and can select the products of their choice. But, there are several drawbacks of the e -commerce websites as well as they do not allow the user to check whether the furniture suits their home, at the time of buying. This sometimes leads to the buying of wrong products which might not suit the home of the user. With the help of the augmented reality technology, the users can convert the captured image of their home and convert it into digital view which allows the users to have a better understanding of which furniture and the electronic equipment might be suitable. There are also several systems which are developed in the similar pretext.

A prior art patent WO2014182545A1 by Masaki Nakadaa and Konstantinos Sideris discloses a Virtual object scaling in augmented reality environment. A computer implemented method, implemented by hardware in combination with software, includes: placing an object in a real-world environment, the object being paper currency having at least one image thereon; using a portable computing device, acquiring an image of the object; attempting to recognize said object using at least one target image; based on said attempting, when said object matches a target image, using said object to determine scale and geometry of the real-world environment; and selecting an image from a catalogue; and rendering an augmented reality based on the real-world environment and a virtual object based on a 3D model corresponding to said selected image.

Another prior art patent US8069095B2 by Elliott Glazer et al. discloses a Virtual reality shopping experience which illustrates a method for a user to shop online in a three dimensional (3D) virtual reality (VR) setting by receiving a request at a shopping server to view a shopping location, having at least one store, and displaying the shopping location to the user's computer in a 3D interactive simulation view via a web browser to emulate a real-life shopping experience for the user. The server then obtains a request to enter into one of the stores and displays the store website to the user in the same web browser. The store website has one or more enhanced VR features. The server then receives a request to view at least one product and the product is presented in a 3D interactive simulation view to emulate a real-life viewing of the product.

Another prior art patent US20020158873A1 by Todd Williamson discloses a Real-time virtual viewpoint in simulated reality environment. In one aspect of the present invention, the inventive system is capable of inserting video images of human being, animals or other living beings or life forms, and any clothing or objects that they bring with them, into a virtual environment. It is possible for others participating in the environment to see that person as they currently look, in real time, and from any viewpoint. In another aspect of the present invention, the inventive system that was developed is capable of capturing and saving information about a real object or group of interacting objects (i.e., non-life forms). These objects can then be inserted into a virtual environment at a later time. It is possible for participants in the environment to see the (possibly moving) objects from any viewpoint, exactly as they would appear in real life. Since the system is completely modular, multiple objects can be combined to produce a composite scene. The object can be a human being performing some rote action if desired. These rote actions can be combined.

The above mentioned prior art patents provides various solutions to the above mentioned drawbacks, but there is no system available through which the user can view the layout of their home through mixed reality, and a system which automatically recommends various furniture and electronic equipment which is suitable, by extracting the information from various e-commerce websites.

Objectives of the Invention The primary object of the present invention is to provide a system and method for automatic furniture and electronic equipment recommendation.

Another object of the present invention is to provide a system and method for automatic furniture and electronic equipment recommendation which is carried out using smartphone.

Another objective of the present invention is to provide a system and method for automatic furniture and electronic equipment recommendation which uses mixed reality along with artificial intelligence system for furniture and electronic equipment recommendation.

Summary of the Invention

The present invention discloses a system and method for automatic furniture and electronic equipment recommendation. The components of the present invention includes a mobile phone particularly a smartphone containing a camera feature, virtual reality feature for creation of 3D view of the captured image, and a mixed reality feature which merges the real and virtual worlds to produce new environment and visualization where the physical and digital objects co-exist and interact in real time. According to the present invention, the mixed reality along with the artificial intelligence is used in image processing where the dimensions of the objects in the captured panoramic image are selected automatically, which are used for prediction.

According to the present invention, the mobile application is used to calculate the dimensions of the object which are present in the captured panoramic image automatically and the similar objects which are related to the object in the captured panoramic image are suggested. The image of the object for example: Fan, AC, Table etc., is used to suggest the alternative of its class to display the similar related objects from website with a competitive price.

The present invention uses the deep learning technique with which the user can train the objects’ dimensions to image clicked in panoramic view. The main idea behind the invention is that the objects having sharp images get selected automatically. With the help of the present invention, the user can use virtual reality feature to shrink or adjust the dimensions of matching object’s image on the website to the panoramic image and finally replace and remove the objects in the processed image. According to the present invention, the captured image is processed automatically, where the user can have option to replace or remove the electronic equipment or furniture in the processed image. The objects can be replaced using the images at websites or shopkeeper.

Brief description of the drawings

FIG 1 illustrates the image captured using the camera of the smartphone.

FIG 2 illustrates the final output of the image processing which was captured using camera, with the selected objects. FIG 3 is the flowchart illustrating the diagram of flow of manual and automated steps.

Detailed description of the Invention

The present invention discloses a system and method for automatic furniture and electronic equipment recommendation. The components of the present invention includes a mobile phone particularly a smartphone containing a camera feature, virtual reality feature for creation of 3D view of the captured image, and a mixed reality feature which merges the real and virtual worlds to produce new environment and visualization where the physical and digital objects co-exist and interact in real time. According to the present invention, the mixed reality along with the artificial intelligence is used in image processing where the dimensions of the objects in the captured panoramic image are selected automatically, which are used for prediction. FIG 1 illustrates the image captured using the camera of the smartphone. Referring to FIG 1, the camera present in the smartphone can be used to capture the panoramic image of the house of the user, where the objects which has to be removed or replaced are located. As illustrated in FIG 1, the objects 1, 2, 3 and 4 are selected automatically once the image is captured. The captured can image can be processed with the help of the missed reality platform of the present invention.

FIG 2 illustrates the final output of the image processing which was captured using camera, with the selected objects. Referring to FIG 2, the captured image can be processed using the mixed reality feature. According to the present invention, the objects having sharp edges in the captured image gets selected automatically i.e. the objects 4, 5, 6 and 7. The selected object 4 can be replaced with other furniture and compatibility can be checked. Also the object 7 can be removed from the place or replaced. The captured image is processed to finally either replace or remove the objects and try the new and similar objects.

According to the present invention, the mobile application can be used to calculate the dimensions of the object which are present in the captured panoramic image automatically and the similar objects which are related to the object in the captured panoramic image are suggested. The image of the object for example: Fan, AC, Table etc., is used to suggest the alternative of its class to display the similar related objects from website with a competitive price.

The present invention uses the deep learning technique with which the user can train the objects’ dimensions to image clicked in panoramic view. The main idea behind the invention is that the objects having sharp edges get selected automatically. With the help of the present invention, the user can use virtual reality feature to shrink or adjust the dimensions of matching object’s image on the website to the panoramic image and finally replace and remove the objects in the processed image.

FIG 3 is the flowchart illustrating the diagram of flow of manual and automated steps. With reference to FIG 3, the process of replacing the existing furniture or electronic equipment or place in blank space of the image or video of a living room or establishment includes: manually clicking for an image of the room or clicking for video mode focusing the living room, which can be achieved through speech or touch on image on a device or smartphone; identification of boundary lines in the image or video by various methods which can be search-based (first order derivative), zero-crossing based (Second order derivative) and phase congruency to provide/calibrate boundaries to objects in image with colors; manual selection of the image of furniture or electronic equipment in the living room which can be achieved through speech or touch on image of device or smartphone, which the customer wishes to replace with or without occlusion; identification of the classification of selected image based on the category of the image, using Deep learning technique to trigger a search of furniture or equipment type with Keywords on web or any other repository of alternative furniture and electronic equipment with dimensions; to calibrate or test the enlarged/optimized of images on a website and repository; conversion of the three dimensional image on phone to two dimensional image and search for most similar image of furniture or electronic equipment the website and repository; converting the brightness at which image on web /repository is clicked to the brightness of room; re-conversion of two dimensional image to three dimensional image to replace or place in the furniture or electronic equipment in the living room; and location of the image or video based on the Inertia Measurement Units (IMU) with GPS for spatial computing or any other methods like machine vision for plane detection and point clouds for depth measurement with lens or split lenses.

According to the present invention, the captured image is processed automatically, where the user can have option to replace or remove the electronic equipment or furniture in the processed image. The objects can be replaced using the images at websites or shopkeeper.

There are several advantages of the present invention. The primary advantage of the present invention is that the captured panoramic image can be easily processed.

Another advantage of the present invention is that the objects with sharp edges get automatically selected.

Yet another advantage of the present invention is that the selected objects can be either removed to obtain a new processed image.

Yet another advantage of the present invention is that the selected objects can be replaced with new objects to obtain a new processed image.

Yet another advantage of the present invention is that the selected objects can be replaced with new objects by taking the images of the objects from e-commerce websites or from the shopkeepers. Yet another advantage of the present invention is that the furniture and electronic equipment can be easily replaced to view a real time processed image of the captured image.

Claims

I Claim:

1. A system for automatic furniture and electronic equipment recommendation, comprising: a. a smartphone containing a camera feature; virtual reality feature for creation of 3D view of the captured image, and a mixed reality feature which merges the real and virtual worlds to produce new environment and visualization; and

b. a mobile application for image processing; wherein, the smartphone and the mobile application uses the mixed reality feature, along with the artificial intelligence for image processing where the dimensions of the objects in the captured panoramic image are selected automatically, which are used for prediction.

2. A system as claimed in claim 1 wherein, the smartphone camera is used to capture the panoramic image or any other image of the living or working place of the user.

3. A system as claimed in claim 1 wherein, the mobile application can be used to calculate the dimensions of the object which are present in the captured panoramic image automatically and the similar objects which are related to the object in the captured panoramic image are suggested.

4. A system as claimed in claim 1 wherein, the mobile application uses deep learning technique with which the user can train the objects’ dimensions to image clicked in panoramic view.

5. A system as claimed in claim 1 wherein, the objects having sharp edges or curved edges get selected automatically.

6. The method of replacing the existing furniture or electronic equipment or place in blank space of the image or video of a living room or establishment, comprising:

a. manually/automatically clicking for an image of the room or clicking for video mode focusing the living place through smartphone;

b. identification of boundary lines in the image or video by various methods;

c. manual selection of the image of furniture or electronic equipment in the living place on smartphone, which the customer wishes to replace with or without occlusion;

d. identification of the classification of selected image based on the category of the image, using deep learning technique;

e. triggering a search of furniture or equipment type with keywords on web or any other repository of alternative furniture and electronic equipment with dimensions;

f. calibration of the enlarged/optimized of images on a website and repository; g. conversion of the three dimensional image on phone to two dimensional image and search for most similar image of furniture or electronic equipment the website and repository;

h. converting the brightness at which image on web /repository is clicked to the brightness of room;

i. re-conversion of two dimensional image to three dimensional image to replace or place in the furniture or electronic equipment in the living room; and

j. location of the image or video based on the Inertia Measurement Units (IMU) with GPS for spatial computing or any other methods like machine vision for plane detection and point clouds for depth measurement with lens or split lenses.

7. The method as claimed in claim 6 wherein, the boundary line identification in the image/ video can be achieved by methods including:

a. search -based method (first order derivative);

b. zero-crossing based method (Second order derivative); and

c. phase congruency method; to provide/calibrate boundaries to objects in image with colors.

8. The method as claimed in claim 6 wherein, the method can dynamically replace a three dimensional image of an object in an image or video with a proposed object from a website which is in a two dimensional or three dimensional form.

9. The method as claimed in claim 8 wherein, the object can be furniture, electronic equipment, or any other object.