WO2014098553A1

WO2014098553A1 - System for the interaction of customized graphic articles in real time by detecting a printed reference marker

Info

Publication number: WO2014098553A1
Application number: PCT/MX2013/000184
Authority: WO
Inventors: Rubén Librado RODRÍGUEZ CHAVANA; Andrés Alfredo SERANGELLI ANDRADE
Original assignee: Rodríguez Chavana Rubén Librado; Serangelli Andrade Andrés Alfredo
Priority date: 2012-12-18
Filing date: 2013-12-17
Publication date: 2014-06-26
Also published as: MX2012015026A

Abstract

The invention concerns a system for the interaction of customized data in augmented reality which can include multiple articles such as invitations, cards, photographs, headstones, museum articles, printed and televisual advertising, art, labels and mascot tags, inter alia. A control module and an interface module are connected by processing data by an image-capturing device, the images being compared with a pre-established marker pattern, which marker is included in the front, rear or inner part of the articles. The marker is detected in the real environment and processed with customized multimedia data in order to display on the screen of a computer, television, cell phone or tablet the image of the product or service which the individual, business or enterprise wishes to display in augmented reality. The individual, business or enterprise can display in augmented reality images of the product or service provided as well as personal data.

Description

PERSONALIZED GRAPHIC ITEM INTERACTION SYSTEM IN REAL TIME BY DETECTION OF A MARKER REFERENCE

PRINTED

DESCRIPTION

OBJECT OF THE INVENTION The present invention relates to the means for different articles or objects which present augmented reality functionality. Depending on the article, augmented reality will display the corresponding information on the screen or in some other medium. So the article will present a real world environment while at the same time it will have a personalized virtual image superimposed on it real world environment.

BACKGROUND

Augmented reality today is divided into two large groups. The first is developed in services based on the positioning of the device and the second group is used under the analysis of an image.

Augmented reality applications that are based on the positioning of the device, usually do not analyze a specific image * rather they use the different sensors that the device has, such as: compass, GPS, gyroscopes, accelerometers, temperature sensors, sensors Photoelectric, among others. Some of the products found today in the market similar to these types of applications are the following:

Layar is an application for cell phones, iphone and android, which uses augmented reality and is presented in Mobile Augmented Reality Applications by D. Prochazka, M. Stencl, O. Popelka and J. Stastnyusa, June 2011, p. 1 - 2. This application takes advantage of the hardware of current cell phones, such as the camera, compass, accelerometers, among other sensors to give the location actually augmented through the cell phone screen and through the internet. By using the information that the sensors give you, the cell phone, you can calculate: the route to the destination, nearby shops, points of interest, tourist places, banks, schools, universities, restaurants, etc. On the other hand, the same document presents MaxSea which is a maritime navigation software for both sailboats or boats. It allows a safe navigation as it anticipates risks at sea and gives you the location through a GPS. The augmented reality is put into practice when the user visualizes through his computer the position of the sailboat in the sea and the distance he needs to reach his destination.

Augmented reality applications that are based on the analysis of an image, must first process that image through a code that may or may not be located on the device and then deliver the final result to the user. For this type of applications it is necessary to have a visual sensor (cell phone camera, webcam, special lenses, etc.), if the code is not stored in the device it is necessary to have an internet connection, so that finally on a screen reality can be perceived increased Document ES2318989 presents an interior decoration system by augmented reality. This system is based on an image that captures the camera and then expands it in the room where you want to place some desired furniture and people can see through augmented reality as said furniture would be inside the room without having bought the furniture . Some of the products that are currently on the market that respond to these types of applications are the following:

Some books include what augmented reality is to improve or reinforce knowledge as presented in Between The Lines by K. Moltenbrey, January 2005, Computer Graphics World, p. 28 - 31. This is achieved by including a grid-shaped image that is the one that recognizes the code, (which can be housed inside a computer or through the internet) through the webcam, and then displays on the screen the element that That image contains. They are also included in different manuals, from cars to aircraft, either to identify parts or to learn how they are assembled. It is planned that technicians will not be able to repair aircraft in the future without having completed a simulation time number through this technology.

Although QR Codes do not belong to the augmented reality group, part of what is behind their interface if it is related. As mentioned in their article in Benchmarking the Use of QR Code in Mobile Promotion by S. Okazaki, H. Li and M Hirose, March 2012, Journal of Advertising Research, p. 102 - 116, the codes that some cell phones recognize through their integrated camera and through the internet, are also some bases that augmented reality includes. QR codes, which mean "Quick Response" in English, began to be used in the automotive industry to track parts of cars. Seeing the potential of this technology began to be used in advertising campaigns. Through these images the cell phone downloads information, music, tones, games, videos, images, etc.

The invitations and business cards, have not changed much compared to those that were invented hundreds of years ago where they are usually single-sided where the relevant information of the person is presented some of them with photographs.

There are also purely virtual invitations or business cards, which are sent through the network. These cards are not physically present although they can be printed, however the real never interacts with the virtual.

The first photographs that came out were those of black and white, over the years they have changed to colors and nowadays they are also digital being stored in some internal or external storage unit.

Pet plates, labels and cards (banks, establishment, identification, etc.) always reveal the owner's information, this information can bring collateral damage if it falls into the wrong hands at the time of a loss or theft.

Print and television advertising can only show an article or service, being impossible to show several products or services in the same place on a page since the images are superimposed when trying to do so. The above is repeated if done in television advertising. In the museums, the artist's information is shown next to the work, the way in which the person sees the data is very monotonous. Also modern art is still seen through a canvas with different techniques.

The tombstones only contain written information about them which mention the person's name, some thought and year of life and death. Augmented reality in the tombstones is something that does not exist and that would allow the person who sees the tombstone to observe some photo, video, sound or text in augmented reality of the deceased person, the present invention also includes this application without being limited to it given The flexibility in its operation.

For this reason I have devised several articles, those mentioned above, that include augmented reality. Augmented reality is one that mixes the real with the virtual, differentiating itself from the purely virtual that only houses tacit elements and does not include the user within the application.

Apart from including augmented reality, invitations and lapids are personalized; opening the possibility of exclusive content for authorized users on a site, that is, augmented reality includes images, videos, text and sounds that people want to include.

This invention is not a simple invitation, card, photograph, tombstone, art, print or television advertising, label or plaque for pets, it goes further “since it introduces the user to interact with augmented reality, allowing the characteristics of these Personalized articles are enriched by the technological resources currently offered by computer science and that allow augmented reality to take place. 201

BRIEF DESCRIPTION OF THE FIGURES FIG. 1 shows the front face of a business card where a marker part of the system of the present invention is included.

FIG. 2: shows the back side of a business card where a marker part of the system of the present invention is included.

FIG. 3: shows a schematic diagram explaining the system of the present invention. FIG. 4: shows an example of a custom marker applicable within the system of the present invention.

FIG. 5: shows an example of the result displayed on the screen applying the system of the present invention.

FIG. 6: The front face of an example tombstone is shown where a marker part of the system of the present invention is included.

FIG. 7: The application of a marker part of the system of the present invention in a photograph is shown.

FIG. 8: The pet plate is shown where a marker part of the system of the present invention is included.

FIG. 9: The front face of the label is shown where a marker part of the system of the present invention is included.

FIG. 10: The front face of an advertising magazine is shown, where ¹ includes a marker part of the system of the present invention. DETAILED DESCRIPTION OF THE INVENTION

Innovation in the present invention is a system that links a ¹ virtual environment with real environment different by using different items which include a marker customizable reference, as shown in FIG.l, said marker is processed by a Image analysis, which detects the marker through an image capture element, either a webcam or interactive glasses, obtaining an image of a real environment (the environment where the user is) and performs a process of Matrix transformation and comparison against a predetermined pattern, detecting the presence and position of the reference marker, subsequently integrates a virtual element (images, text, videos, sounds) that is in a database linked to an application (invitations, cards, labels, pet plates, photographs, museums, art, tombstones and containing personalized information) in such a way that through an item Visual display (laptop screen, computer, smart TV, smart phone, or personal tablet). A dynamic result of the integration of the real environment and overlapping virtual elements is obtained. The reference marker will preferably be defined black and white in two or three dimensions, as long as the periphery of the reference is defined; At the back of the marker it is not required to add any component, as seen in the application on a business card illustrated in FIG. 2.

The explanation of the operation of the interaction system of personalized graphic articles in real time by means of the detection of a reference marker printed invention can be seen graphically in the diagram of FIG. 3. Also the size of these items may vary depending liked by ¹ person. The front face, as can be seen in FIG. 1, the invitation will depend on the person, being able to include: only event information, logo or a combination of both as well as the marking or image "L"; On the back of the invitation (refer to FIG. 2) it will contain the company logo in conjunction with the name and slogan, instructions for the use of the cards, contact for technical support and relevant information regarding to the card. The text that is included in the cards will be mainly in Spanish or English although, as such cards will be personalized, they can be in any language depending on the person.

The invitations cover a wide range of events:

• Birthday.

• Fifteen years.

• Weddings.

• Social events.

The cards cover a wide range of private, public and personal sector establishments:

Bank cards: debit or credit

Retail cards, wholesale, department stores and markets.

Identification Cards, such as: Driver's license, citizen identification.

Conference cards, entry and exit of staff to an establishment. The tombstones cover a wide range of places:

• Holy Field.

• Cemeteries

Pet plates are understood to be for all types of macota.

The code, where the program was written, is based on Microsoft Visual Studio 11 Beta and developed under the programming environment in C #, although this code could be exported to any programming environment. Part of the code describes the interface with which the user will be interacting, said interface consists of at least one button which allows the user to initiate communication with his camera (webcam) for the exchange of information between the surrounding environment and the space virtual.

In order for a patent to be granted, the product that is trying to be protected has to have an industrial application, it is detailed in detail how the product was created step by step in the following points.

The real-time personalized graphic article interaction system by detecting a printed reference marker works through a series of stages which are described below:

A. A virtual interface module is generated which will be presented in an on-screen display element to the user and that will be where the interaction with the virtual elements takes place.

B. A control module linked to the interface module, initiates an image capture device by manual or automatic actuation, said feature is customizable, within the interface module and such a module control should have access to the database where the digital information of a preset pattern of the reference marker is stored and ate the personalized multimedia content to use.

C. The image capture device receives the information from the real environment including the reference marker and, within the control module, said information is processed and transformed to a binary file, detecting the vertices of the reference marker in the image performs a marricial transformation of the preset pattern of the reference marker, they are compared in such a way that the position and orientation of the reference marker printed on an object is detected and located, such an action is carried out in real time.

D. Once the reference marker printed on an object in the real environment has been detected and after it has been compared with the preset reference marker pattern; The control module will link the personalized multimedia information ¹¹ , displaying on screen in real time the result of the integration of virtual elements with real environment, also adjusting the multimedia content to the position and orientation of the reference marker printed on the object, performing the execution of this task continuously.

Example of embodiment of the invention

A virtual interface module is generated which will be presented in an on-screen display element to the user and that will be where they interact with the virtual elements. You need to have Microsoft Visual Studio (at least the 2010 version), with the Silverlight 4 Tools extension for Visual Studio and have A standard webcam. However, it is not limited to such programming package or device.

A new Visual Studio project is started and a new Silverlight application is created.

Two files MainPage.xaml and MainPage.xaml.es are opened. In the lainPage.xaml the design of our application is specified. This design may change depending on the occasion that our personalizable article will be used. If it is going to be used for pet plates, this page is decorated with different pets so that the user feels identified with the product.

In this same page MainPage.xaml the augmented reality will be presented. The video stream of the webcam is displayed inside a rectangle with the ^■ ■ help of a VideoBrush. The button is used to ask the user for access to the webeam and start the capture. The image that is presented in augmented reality is the customizable, the person chooses which image or images they want to appear in augmented reality. These images can be found in any valid format such as: .jpg .png .jpeg .pdf .gif among many more. These images are then uploaded to the project using the following instruction:

Visibility = "Collapsed"

<Image x: Name = "heart" Width = "300" Height = "300"

Source = "Assets / PERSONALIZED IMAGE.png" Canvas.Left = "335"

Canvas.Top = "- 435" l>

In the MainPage.xaml.es file, the codes to initialize the webcam are written: captureSource = new CaptureSource ();

captureSourceNideoCaptureDevice

CaptureDeviceConfíguration.GetDefaultVideoCaptureDevice = ();

6. Load the marker pattern (in this case the pattern that is loaded is from the "L" marker, of FIG. 4 although this may change to any other)

arDetector = new CaptureSourceMarkerDetector ();

var = Marker.LoadFromResource marker ("MarkerJLpat", 16, 16, 80);

7. Matrix transformation is performed

var vp = Matrix3DFactory.CreateViewportTransformation ^

Viewport.ActualHeight);

var m = Matrix3DFactory.CreateViewportProjection (world, Matrix3D.Identity, arDetector.Projection, vp); heart.Projection = new Matrix3DProjection {ProjectionMatrix = m};

8. The perspective projection is calculated with the plane closest to 1 and the plane far away from 5000

arDetector.Initialize (captureSource, 1, 5000, marker);

9. Marker detection, the event is activated if at least one marker was detected If (marker! = False)

{

// show the custom image }

else

// It shows nothing Example of use of the invention

1. Enter the website where the personalized information (text, images, videos, sound) with augmented reality is located. In this website you can customize your image or if it is already personalized you can go

i

directly to the link provided.

2. Initialize the user's camera. Ask the user if they want to initialize communication, if so, continue with step 3. Otherwise skip to step 10.

3. The rectangle corresponding to the camera goes blank and starts transmitting video.

4. The default image "L" (FIG. 4) is loaded into the program to start comparing it with the one on the physical card.

5. When the camera captures the image inside the user's card, it transforms into black and white to identify the vertices and corners.

6. Perform a matrix transformation so that the image "L" is; view from a camera plane.

7. Make a comparison between the image "L" that the camera is capturing and the image "L" that is predefined in the program. 8. If the two are the same, display the personalized information (image, video, text, sound). Otherwise omit the result and return to step 4.

9. The user enjoys augmented reality.

10. The program ends.

The personalized information that will be included in the code described above, will depend on the taste of each person. These images, videos, text and sounds will be processed by editing programs of different formats to give them some type of design before being included in the detailed code. In one embodiment, several images were included which were placed in collage and containing an informative text of said products, see FIG. 5. The figures or designs that can be made with personalized photos are unlimited as it will be depending on the person's taste. The included text - within custom images can be in any language. After the custom image is finished, it is exported to the program in Microsoft Visual Studio 11 Beta so that it can be viewed in augmented reality.

The materials used in the invention of these items in augmented reality, will be to the client's taste in terms of their dimensions, shapes, colors, designs and accessories.

Examples of application of the invention

The present invention can be applied to countless elements and materials, without being limited to the following, preferential uses are presented: invitations, cards, labels, Advertising and pet plates will be made of art paper, bond, Kraft, corrugated, gloss, matt lamination, or 200GSM / 230GSM / 250GSM / Cl S / ÍC2S / woodfree / special paper, aluminum, metal, plastic, glass , as well as recycled material depending on the client's taste. The material of the tombstones can also vary from plastic, glass, wood, brick, cement, metal and among others.

In one embodiment, the invitation was exposed along with the image "L" to the webcam and the custom image appeared as seen in FIG. 5. For this occasion the personalized image in augmented reality.

In another embodiment a different invitation was used with different materials and dimensions FIG. 1 in which the process that was followed was the same as the previous one. The invitations that could be used with augmented reality, can range from one or several sheets, type, shape, material and color according to the person's desire.

In a following embodiment, the advertising magazine (FIG. 11) was shown with the image "L" to the webcam and an advertising image for a product appeared. The same figure in "L" was shown to the webcam with a different league of internet and a different advertising image was appreciated than the previous one.

In one embodiment, an invitation was made for a wedding measuring 15 _cm Width x 25cm Height containing welcome and wedding information, in addition to including the "L" marker that will be used by the program to display personalized information actually augmented in In this case there are several images of the wedding, an image that shows the place of the event, another image that shows the bride and groom and a last image that shows relevant information about the event, such as: date, place and time. Although this last It could also be text, video and sound. Instructions for use are found on the back of card ¹ . The size, material and text inside the card may vary.

In a different embodiment, a photograph of graduates along with the image "L" was exposed to the webcam and the custom image appeared as seen in FIG. 7. For this occasion, the personalized image in augmented reality was decided to be in the form of a square showing a photo of the augmented reality generation.

In a later embodiment, a pet plate was exposed along with the image "L" to the webcam and the personalized image with text appeared as seen in FIG. 8. For this occasion the personalized image is of a dog with the personal data of the owner to whom it belongs.

In a last embodiment, the marker of the tombstone (FIG. 6) was exposed to the webcam and a personalized video appeared in augmented reality of the deceased person. This video shows the person saying a few words to those who visit the tombstone. '

Claims

CLAIMS Having described my invention sufficiently, I consider as a novelty and therefore claim as my exclusive property, the content of the following clauses:

1. A system of interaction of personalized graphic articles in real time by detecting a printed reference marker characterized in that it works through a series of stages which are described below:

B. A control module linked to the interface module, initiates an image capture device by manual or automatic actuation, said feature is customizable, within the interface module and such control module must have access to the database where The digital information of a preset pattern of the reference marker is stored as well as the personalized multimedia content to be used.

C. The image capture device receives information from the real environment

i

including the reference marker and, within the control module, said information is processed and transformed to a binary file, detecting the vertices of the reference marker in the image performs a mairicial transformation of the preset pattern of the reference marker, they are compared in such a way that detects and locates position and orientation of the reference marker printed on an object, such action is carried out in real time. D. Once the reference marker printed on an object in the real environment has been detected and after it has been compared with the preset reference marker pattern; The control module will link the personalized multimedia information, displaying on screen in real time the result of the integration of virtual elements with real environment, also adjusting the multimedia content to the position and orientation of the reference marker printed on the object, performing the execution of Said task continuously.

The interaction system of personalized graphic articles in real time by means of the detection of a characterized printed reference marker, according to claim 1, characterized in that it is applicable to any means capable of displaying the reference marker printed in a defined manner without being limited to the condition of two dimensions or object material.