ES2350790B1 - PAAR TEAM VISUALIZATION OF VIRTUAL AND / OR MULTIMEDIA SUPERPOSED INFORMATION ON AN IMAGE OF THE REAL ENVIRONMENT - Google Patents

Abstract

Equipment for viewing virtual and / or multimedia information superimposed on an image of the real environment, in a position and orientation calculated in real time by various devices included in said equipment, comprising a base platform (1), anchored to the ground, which supports the user who interacts with the system as well as all the elements that make up the visualization equipment integrated in a column (2); which can rotate on itself 360º and has a device for controlling the horizontal angle of rotation and automatic control of the angle of inclination of an image capture camera located in a structure (3), located on this column (2), located above it and fixed in it so that they rotate in unison.

Description

Equipment for visualization of virtual and / or multimedia information superimposed on an image of the real environment.

Object of the invention

The invention consists of a device that allows the visualization of all types of virtual information created with a computer (three-dimensional models, texts, images or videos) positioned, oriented and scaled according to the point of view determined by the user, on a video image of a real environment , allowing him the possibility of observing, for example, a virtual three-dimensional model of the reconstruction of what an archaeological site was originally, overlapping on the real image of the current state of the site. It also allows the reproduction of audio files, which it is activated automatically based on certain actions performed by the user.

Background of the invention

Currently there are multimedia information visualization systems based on virtual reality, where the whole scenario is tricky, that is, it is an immersive technology, in which the user has no references about the real environment around him.

Systems known as augmented reality are also known, in which real-world images are superimposed with virtual elements designed by computer. This technology offers a wide range of possibilities in areas as diverse as culture, medicine, education, tourism and real estate development, interior decoration or industry.

In augmented or composite reality systems, the user can interact with virtual training that mixes with the real world in a natural way. In addition, the vision of the outside real world is complemented by the superposition or composition of three-dimensional virtual objects, audio, video, text, flat images, and in general any kind of multimedia information that allows to recreate an enlarged "reality" of the real world.

Augmented reality has had an important development thanks to the works published in 1999, under the title “Recent Advances in Augmented Reality”, by the doctors: Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simón Juliery Blair Maclntyre.

In the patent literature, document WO-2006112743 describes a visualization device based on this technology, which allows to improve the virtual tourist visit of historical or scenic places of another type, adding multimedia elements to the real landscape and composing a combined image that represents the user. This system includes a single solution that incorporates: remote local databases; mechanisms that measure the position and orientation of the system; a system to capture real images through a video camera; a computer for processing the image and composition of virtual elements, and a presentation screen for the final image. This system has two sensors that determine the position of the camera or move to the focus: a rotation sensor, connected horizontal distance, and another that measures the angle of inclination of the video cameras that provide the real images of the scene.

Description of the invention

The invention proposes an improved system or equipment that allows multimedia viewing independently, or combined with the reality of the scenario. Unlike the previous system, in which the user moves the camera manually and consequently it is necessary to determine the position towards which it focuses, in this case , there is no sensor to measure the vertical position, and that the camera tilts up and down controlled by the software of the equipment depending on where the user indicates, when the process is reversed, that equipment controls where the camera focuses and the exact position of the same according to the commands entered by the user. The same happens with the camera's zoom, it is operated from the device and not in the camera itself, with which, the program that controls it knows the set zoom perfectly. These operations are performed by the user through buttons that you can press through the touch screen.

The equipment of the present invention comprises:

a) A base platform, anchored or supported on the ground, which serves as a support for the user who interacts with the system. In large scenarios, such as archaeological sites, the availability of a space for a group of observers has also been planned. This platform also supports all the elements that make up the visualization equipment that are integrated in a column or vertical orientation structure.

b) The aforementioned vertical column is either articulated on the platform of the base, or rotates in its upper half over the lower half. The rotation that can be made is 360º and can be infinitely turned, without the need to establish a predefined orientation. The turn on itself is done manually by the user, its axis being associated with a rotation angle control device, consisting of a magnetic rotation sensor that, connected to an electronic control board, determines and transmits to the system, in real time, the horizontal angle of the column which has been moved to the column when the user turns in any way.

This column integrates the various hardware devices that make up the equipment: a central processing unit (CPU), a touch screen in which the real image of the environment is displayed and / or a composite image based on the superposition of positioned and objectively oriented virtual objects on the real image of said environment, as well as the orientation detection devices that determine the horizontal rotation angle of automatic control of the inclination angle of the image capture chamber located on this column, an audio reproduction device, as well as a power supply of all These devices and a cooling and / or heating and / or ventilation system for them.

c) An image capture camera placed on a structure, located above the rotating column and fixed in it so that they rotate in unison, which has a tilt mechanism up / down, operated by a stepper motor, controlled by the CPU system that sends the commands necessary to mobilize the camera, tilting it up or down and, consequently, can determine its orientation at all times, in conjunction with the device that determines the horizontal rotation of the column and the zoom factor applied to the camera, calculating the focus on the focused area, as well as the degree of magnification of the real image and virtual information scaling.

Description of the fi gures

To complement the description that is being made and in order to facilitate the understanding of the features of the invention, a set of drawings is attached to the present specification in which, for illustrative and non-limiting purposes, the following has been represented:

The figure 1 shows the views seen in the plant, elevation and profile of the display equipment according to the invention.

Figures 2 and 3 correspond to perspective views, respectively from the stage-oriented area and from the place where the user is placed on the platform (1).

Preferred Embodiment of the Invention

The elements that make up the system are described in more detail below:

A real-time image capture device that obtains information from the first scene, that is, from the real environment. Said device is any type of video camera, which optionally includes a lens with motorized variable focus and an electronic board that allows to control the zoom, focuseirisa through a CPU, or a fixed focal lens.This element is located in a structure (3) which is a structure (6) of tubes, links with the main body of this equipment consisting of the column (2).

A platform on which the image capture equipment is placed, which is mechanically connected to a stepper motor and is mounted on a controller board, which is controlled by the PC, allows the sending of commands to tilt the platform on which the camera is located above or below. The platform that supports the camera, together with the stepper motor and the control electronics, is located in the same structure (3) that uses the image capture device.

A rotation device formed by a magnetic type rotation sensor, connected to a plate with the control electronics necessary to determine and transmit to the CPU, the data of the horizontal rotation angle that the physical device has moved through the manipulation of the handle by the user. the column (2), or rotates on the base platform (1), or an intermediate plane as shown in the figures 2 and 3. The user is the one who manually rotates in the horizontal plane of the upper part of the column (2), by means of a steering wheel or handle (5), preferably located under the screen (4) in which virtual and / or real images are represented.

A computer or a CPU with a series of peripherals, including a graphics processing unit, in which the application software is executed, in which the camera tilt control, located in a structure (3), and in which it is performed is performed communication with the turning system.

A software application that is formed by a graphical window interface, in which a main menu can be accessed, through which the language can be selected and the device's operating mode: conventional viewing mode, multimedia mode and composite viewing mode with composite reality. In the conventional viewpoint modes and with composite reality, the software application allows the user, as a minimum functionality , zoom more or less in the image of the real scene and / or in the image of the composite scene (real image complemented by virtual objects), as well as tilt the camera up or down. The user interacts with the application by pressing certain buttons of a graphic application using a touch screen (4). In multimedia mode, the device acts in a way that the software application is in charge of offering the user the possibility of viewing images, videos and / or listening to audio files related to the location in which the device is located.

A central processing unit (CPU) that must be configured to generate specific multimedia content (modeling in 3D, text, image) for each orientation and inclination of the detected camera, as well as to create a second scene as a result of the superposition of the multimedia object on the first scene in a particular position for the global positioning and orientation system, so that you select to offer the user the experience of mixed or compound reality.

A screen (4), as an output device, which allows the user to appreciate the mixed reality, that is, the display of the image of the composite scene (in the case of the operating modes of the conventional viewpoint and mixed reality viewer) ), by means of the video signal of the real scene on which the virtual information positioned and directed at each moment is superimposed depending on the point of view of the camera determined by the user. This point of view is obtained from the data obtained from the global positioning and orientation calculation system that includes the system.

A global device for calculating the positioning and orientation of the user's point of view, that is to say, obtaining the exact point in which the camera is captured in each video picture. It is available to take care of obtaining the orientation and positioning data that will be used to calculate the projection matrix of the system, determined based on the rotation performed by the user, and the inclination of the camera applied.

The user can interact with the system in different ways:

-

Through the touch screen (4) to select different modes of operation of the application.

-

Through the touch screen (4) to select different languages for the operation of the application.

-

Through the rotation of the column (2), by manipulating it (5), it has integrated device on the outside. In this way, the user rotates the device horizontally, selecting the field of vision that the camera captures in a 360º range.

-

Through the touch screen to tilt the camera up or down, so that the field of view of the camera is selected.

-

Through the touch screen (4) to zoom more or less in the image of the real scene and / or in the image of the composite scene (real image complemented by virtual information).

-

Through the touch screen to select or activate different layers of virtual information on the composite image such as: three-dimensional models, texts, images, videos, audio, etc.

-

With any type of graphic interface button, through the touch screen, for auxiliary functionality that complements the previously described points.

It is planned to provide this equipment with an anti-vandal trackball or mouse, hidden under a plate so that it remains inaccessible to the user whenever and when there is a malfunction of the touch screen.

Likewise, it has been foreseen that the database containing the multimedia information can be located locally on the computer included in the device or it can be located on an external server and decentralized from the device.

The rotation device can also be composed of a magnetic rotation sensor of horizontal type or any other element of measurement of the horizontal angle of rotation made. On the other hand, the tilt mechanism of the upper or lower chamber can be implemented by a stepper motor and a controller to control these movements through the PC or can be solved by any other type of electro-mechanical solution that provides the same result.

The image capture device can be a single block that includes a camera, motorized or non-motorized lens, with variable focus or with a fixed focus and controller to handle zoom, focus and iris from the CPU. It can also be made up of totally blocks independent for the camera, lens and device

controller. Likewise, the image display device can be composed of three parts: screen, vandal-proof glass sheet and vandal-proof. Likewise, it can be composed of two parts: touch screen with a thickness such that the use of vandal-proof protective glass is not required.

The device includes an Internet connection card, so that the device software can be monitored from a remote PC for maintenance, testing or statistics collection regarding ways of using the device by the user.

Once the nature of the invention has been sufficiently described, as well as an example of a preferred embodiment, the appropriate effects that the materials, shape, size and arrangement of the elements described can be modified may be recorded, provided that this does not imply an alteration of the essential characteristics of the invention claimed below.

Claims (5)

1. Equipment for visualization of virtual and / or multimedia information superimposed on an image of the real environment, in a position and orientation calculated in real time by various devices included in said equipment, characterized in that it comprises: a) a base platform (1), anchored to the ground, that supports the user who interacts with the system and / or a group of observers, as well as all the elements that make up the visualization equipment integrated in a column (2) ; b) a column (2) that can rotate on itself 360º when operated by the user by means of a handwheel or handle (5), which is associated with a rotation angle control device and integrates the various hardware devices that make up the equipment: a central processing unit (CPU), a touch screen (4) in which the real image of the environment is displayed and / or a composite image based on the superposition of virtual objects positioned and oriented consistently on the real image of said environment, two orientation detection devices that determine the horizontal rotation angle of automatic control of the inclination angle of the image capture camera located in a structure (3), located on this column (2), an audio reproduction device, as well as a power supply of all these devices and a cooling and / or heating and / or ventilation system for them; and c) an image capture camera located in a structure (3), communicated by another structure (6) detubosala column (2), located above this and fixed in the form that rotates in unison, which has a tilt mechanism up and / or down driven by a stepper motor controlled by the application implemented in the equipment.

2.

Equipment, according to claim 1, characterized in that it integrates a device for determining the positioning and orientation of the camera's viewing point, depending on the rotation made by the user of the column (2), consisting of a magnetic rotation sensor, connected to an electronic control platform that determines and transmits to the CPU the horizontal rotation angle that has moved column (2) when the user turns it in any direction.

3.

Equipment according to claims 1 and 2, characterized in that it integrates a device for determining the inclination of the camera, which is moved by a stepper motor, driven from a controller board, which through the CP allows the sending of commands to tilt it up or down and consequently determine instantaneously its orientation for, in In conjunction with the device that determines the rotation made by the column (2) and the zoom factor applied in the camera (3), determine the attitude of the focused area, as well as the magnification of the real image and virtual information scaling.

Four.

Equipment according to the preceding claims, characterized in that the central processing unit (CPU) integrates an application that:

a) carry out the control of the inclination of the chamber and the communication with the rotation of the column (2) as well as the calculations necessary to determine the area focused by said camera for each orientation and inclination of the detected camera, b) creates a scene that is the result of the superposition of a virtual and / or multimedia fi gure on a first real scene of a position determined by the global positioning and orientation system, offering the user a mixed or composite reality of both scenes, and c) includes a graphical interface that allows the user to interact with the application by pressing on a touch screen (4) and / or a mouse and, among other functions, select the operating mode of the device: conventional viewpoint, multimedia or viewpoint with compound reality 6 7 8 SPANISH PATENTS AND BRANDS OFFICE Application no .: 200930333 SPAIN Date of submission of the application: 06.19.2009

REPORT ON THE STATE OF THE TECHNIQUE

Priority Date: 00-00-0000 00-00-0000 00-00-0000

51 Int. Cl.:

See additional sheet

 RELEVANT DOCUMENTS  

Category

56 Documents cited Claims Affected

X

WO 2006112743 A1 (YDREAMS INFORMATICA S A; NABAIS NOBRE EDMUNDO one

MANUEL) 26.10.2006, paragraphs 8.29,30,31,39,43,47,61,63-65,71,72,

74.75; claims; figures.

TO

JP 9096766 A (FUJI PHOTO OPTICAL CO LTD) 08.04.1997 one

TO

PT 103625 A (YDREAMS INFORMATICA S A) 06.30.2008 one

Category of the documents cited X: of particular relevance Y: of particular relevance combined with other / s of the same category A: reflects the state of the art O: refers to unwritten disclosure P: published between the priority date and the date of priority submission of the application E: previous document, but published after the date of submission of the application

This report has been prepared • for all claims □ for claims no:

Date of realization of the report 21.09.2010

Examiner Mª C. González Vasserot Page 1/4

Application number: 200930333 REPORT ON THE STATE OF THE TECHNIQUE CLASSIFICATION OBJECT OF THE APPLICATION  G02B 23/18 (2006.01) G07F 17/02 (2006.01) H04N 13/00 (2006.01) G06F 3/048 (2006.01) Minimum documentation sought (classification system followed by classification symbols) G02B, G07F, H04N, G06F Electronic databases consulted during the search (name of the database and, if possible, search terms used) INVENTIONS, EPODOC, WPI  WRITTEN OPINION Application number: 200930333 Date of Completion of Written Opinion: 09.21.2010

Statement

Novelty (Art. 6.1 LP 11/1986)

Claims 1-4 YES

Claims _____________________________________

NO

Inventive activity

Claims 2-4 YES

(Art. 8.1 LP11 / 1986)

Claims 1 NO

The application is considered to comply with the industrial application requirement. This requirement was evaluated during the formal and technical examination phase of the application (Article 31.2 Law 11/1986).  Opinion Base.- This opinion has been made on the basis of the patent application as published.  WRITTEN OPINION Application number: 200930333 1. Documents considered.- The documents belonging to the state of the art taken into consideration for the realization of this opinion are listed below.

Document

Publication or Identification Number publication date

D01

WO 2006112743 A1 10.26.2006

D02

JP 9096766 A 08.04.1997

D03

PT 103625 A 06.30.2008

2. Statement motivated according to articles 29.6 and 29.7 of the Regulations for the execution of Law 11/1986, of March 20, on Patents on novelty and inventive activity; quotes and explanations in support of this statement Document D1 is considered the closest in the state of the art to the requested invention. In relation to independent claim 1, document D1 describes a device for displaying virtual information and / or multimedia superimposed on an image of the real environment, in a position and orientation calculated in real time by various devices included in such equipment, see paragraph 29 of document D1, which comprises: a) a base platform (1), observe paragraphs 61, 65 and claim 3 anchored to the ground, which supports the user that interacts with the system and / or a group of observers, as well as all the elements that make up the team visualization integrated in a column (2); b) a column (2) in D1 is Column 04, see also claim 2, which can rotate on itself 360 ° when operated by the user by means of a steering wheel or handle (5), has a handle: look at paragraphs 30, 61 and 63 of D1, which is associated with a device of control of the angle of rotation and integrates the various hardware devices that make up the equipment: a central processing unit (CPU) in paragraph 61.75 of D1, a touch screen (4) in paragraphs 29, 30.64.65.74 and claim 8 of D1, wherein visualize the real image of the environment and / or a composite image from the superposition of positioned virtual objects and coherently oriented on the real image of said environment, two orientation detection devices, read the paragraphs 8.29,30,31,43, 47,65 of D1, which determines the horizontal turning angle and automatic control of the inclination angle of the image capture camera located in a structure (3), located on this column (2), a device for audio playback in paragraph 39 of D1, as well as a power supply of all these devices and a system of cooling and / or heating and / or ventilation therefor; Y c) an image capture camera in paragraphs 29,31,43,47,61,64,65 and 72 and claims 1,3 of D1, located in a structure (3), communicated by another structure (6) of tubes to the column (2), located above it and fixed therein which rotate in unison, which has a tilt mechanism up and / or down driven by a stepper motor, see paragraph 71 of D1, controlled by the application implemented in the equipment. The difference between document Dl and claim 1 is that in document D1, Column 04 does not rotate, rotate the chamber, without the Username. The effect of obtaining an image is the same either by rotating the camera with the column (as in the application) and rotating the camera independently of the column (as in document D1), in both processes the same is achieved Outcome. In both cases it is the user who controls the camera. Both devices solve the same technical problem of visualization of virtual and / or multimedia information superimposed on an image of the real environment, in a position and orientation calculated in real time by various devices. A person skilled in the art would use document D1 (Paragraphs 8, 29, 30, 31.39.43, 47.61.63-65.71.72.74.75 and claims 13.8) without making use of the activity inventive and therefore claim 1 does not imply inventive activity (Article 8 LP). The remaining claims do have novelty and inventive activity since no documents have been found with sensors that measure the rotation of the column. Report on the State of the Art (Written Opinion) Page 4/4