CN103777757A - System for placing virtual object in augmented reality by combining with significance detection - Google Patents

Abstract

A system for placing virtual objects in augmented reality combined with saliency detection, including a capture image frame module for reading static images, a 3D object generation module for generating virtual objects, and a display module for displaying virtual-real combined images , is also provided with: a tracking identification module, which is used to perform saliency detection on each frame of image collected by the image frame acquisition module and can quickly generate a saliency map of each image frame; an image processing module, which is respectively connected to the image frame acquisition module , a tracking recognition module, a 3D object generation module and a display module, which are used to complete the placement of the virtual object in the real scene based on the saliency detection. For an unfamiliar scene or an image, the present invention can automatically adjust the size and position of the virtual object, fully considering the visual attention mechanism of the human eye, so that the generated augmented reality scene can be dynamically changed, making up for the current method of placing virtual objects. Some deficiencies improve the visual experience of the observer.

Description

A system for placing virtual objects in augmented reality combined with saliency detection

technical field

The invention relates to a system for placing virtual objects in augmented reality. In particular it relates to a system for placing virtual objects in augmented reality combined with saliency detection.

Background technique

Saliency detection technology and augmented reality technology are involved in image processing. in:

Saliency detection techniques:

The attention and understanding of a scene by the human visual system is determined by a series of potentially different factors, among which the saliency of objects in an image is an important decisive factor. In image analysis and understanding, the salient area of the image is usually only a small part of the image, while the image background occupies a larger part of the image. The so-called salient area can be understood as the main target in the image, which means that human vision can focus on a certain area in the image that can stimulate people's interest in a short period of time. Saliency detection technology is to use computer to analyze images, simulate the attention mechanism of the human eye described above, and calculate and extract salient areas in an image. In the late 1990s, image salient feature analysis based on biological perception began to emerge, and gradually became the focus of research in the field of biological visual perception. This method is based on the human visual attention mechanism, combined with the theoretical knowledge of human psychology and physiology, and simulates the human eye function to establish an image saliency extraction model. This method is driven by biological and physiological mechanisms and based on obtaining a variety of early visual features. A representative algorithm is the Itti algorithm.

Augmented Reality:

Augmented Reality (AR for short) is a new technology developed on the basis of virtual reality, also known as mixed reality. It is a technology that increases the user's perception of the real world through the information provided by the computer system, applies virtual information to the real world, and superimposes the virtual objects, scenes or system prompt information generated by the computer on the real scene, so as to realize the enhancement of reality , At the same time, since the connection with the real world is not cut off, the interaction method is more natural. Augmented reality uses computer graphics technology and visualization technology to generate virtual objects that do not exist in the real environment, and accurately "places" the virtual objects in the real environment through sensing technology, and integrates the virtual objects with the real environment with the help of display devices, and A new environment with real sensory effects is presented to the user. Therefore, the augmented reality system has the new characteristics of combination of virtual and real, real-time interaction, and three-dimensional registration.

Augmented reality is now widely used in mobile applications. This type of mobile application not only allows users to see the real world, but also can see virtual objects superimposed on the real world. It combines the real environment with the virtual environment. a system.

Today, augmented reality technology has been gradually applied to military, medical, construction, education, engineering, film and television, entertainment and other fields. Augmented reality technology simply means applying virtual information to the real world, superimposing computer-generated virtual objects, scenes or system prompts on the real scene, integrating the virtual objects with the real environment with the help of display devices, and presenting them to the audience. The user has a new environment with realistic sensory effects, thereby realizing the enhancement of reality.

Most of the existing augmented reality technologies and related applications use the following two methods when placing virtual objects in a real scene:

The first method: the virtual object is placed on a pre-set special target position in the scene, such as on the hand of the person in the scene to be enhanced or on a fixed pattern appearing in the scene.

The second method: placing a virtual object in a fixed place of the display window, such as placing a virtual prompt or a 3D model in the upper right corner of the display window.

The above two methods have their own suitable application fields, and have achieved good results in some special occasions. However, in some scenarios where augmented reality needs to be applied, the two methods also have their own shortcomings. When using the first method to place virtual objects, if there is no pre-set target object or pattern in the real scene, or the detected target or pattern is not clear enough, the virtual object will not appear in the real scene when displayed On the other hand, when detecting a predetermined target in a real scene, such as a human hand, the technology for detecting and identifying the target is often more complicated, and sometimes the calculation time is longer, which affects real-time performance. When the preset target changes, it will also False detection or missed detection will affect the effect of augmented reality. When the second method of placing virtual objects is adopted, since the virtual objects are in a fixed position of the display window, on the one hand, the virtual objects may block some important content in the real scene and affect the observation effect; on the other hand, due to the location of the virtual objects, It has been fixed, so when the real scene changes, the virtual object does not change with the real scene, and there is no virtual-real interaction effect, so it cannot produce a good effect of combining with the real scene.

Contents of the invention

The technical problem to be solved by the present invention is to provide an augmented reality system that fully considers the characteristics of the attention mechanism of human vision, adopts a relatively fast saliency detection method, and automatically performs the adjustment and placement of virtual objects combined with saliency detection. A system for placing virtual objects in .

The technical solution adopted by the present invention is: a system for placing virtual objects in augmented reality combined with saliency detection, including an image frame acquisition module for reading static images, a 3D object generation module for generating virtual objects, and The display module for displaying virtual and real combined images is also provided with:

The tracking recognition module is used to perform saliency detection on each frame of image collected by the image frame acquisition module and can quickly generate a saliency map of each image frame;

The image processing module is respectively connected to the acquisition image frame module, the tracking recognition module, the 3D object generation module and the display module, and is used to complete the placement of the virtual object based on the saliency detection in the real scene.

The saliency detection performed by the tracking recognition module is realized by using the saliency detection model of IIIt.

The processing step of described image processing module is,

1) First calculate the ratio of the number of pixels in the salient region detected by the tracking and recognition module to the number of pixels in the entire image;

2) Adjust the size of the virtual model generated by the 3D object generation module according to the calculated pixel ratio;

3) Referring to the position of the salient area detected by the tracking and identification module, placing the virtual object at the position corresponding to the salient area in the image captured by the image frame acquisition module;

4) Send each frame of image where the virtual object is placed to the display module for display.

When the display module is displaying, if the virtual object is expected to get the most attention of the observer, the image processing module will adjust the virtual object to the same size as the salient area, and superimpose the virtual object on the real scene. The position of the area is then sent to the display module for display.

When displaying by the display module, if it is desired that the virtual object does not affect the main content of the image, the virtual object is placed next to the salient area or at a certain distance from the salient area and then sent to the display module for display.

During the processing of the image processing module, when the size or position of the salient area of the image detected by the tracking recognition module changes, the image processing module repeats steps 1) to 4) according to the change of the salient area to dynamically adjust the size of the virtual object As well as the location, it has achieved the effect of real-time combination of virtual and real.

A system for placing virtual objects in augmented reality combined with saliency detection in the present invention can automatically adjust the size and position of virtual objects for an unfamiliar scene or an image due to the combination of saliency detection methods, Fully considering the visual attention mechanism of the human eye, the generated augmented reality scene can be dynamically changed, making up for some shortcomings of the current method of placing virtual objects, and improving the visual experience of the observer, especially in the application of placing advertisements. Outstanding advantages, using different placement strategies, can make the placed virtual advertising model attract the greatest attention of the observer, or place advertisements without affecting the observer's viewing of the main content of the scene, and play a role in publicity. Furthermore, the strategy of placing the 3D model in this method is completely automatic and does not require manual participation.

Description of drawings

Fig. 1 is a schematic diagram of the system configuration of the present invention.

in the picture

1: Capture image frame module 2: Tracking and identification module

3: 3D object generation module 4: Display module

5: Image processing module

Detailed ways

A system for placing virtual objects in augmented reality combined with saliency detection according to the present invention will be described in detail below with reference to embodiments and drawings.

A system for placing virtual objects in augmented reality combined with saliency detection of the present invention is proposed in combination with existing saliency detection technology and augmented reality technology: in a given scene or image that needs augmented reality , first simulate the human visual mechanism for saliency detection, and then use the detected saliency area as a benchmark to automatically place and adjust the virtual object according to the specific effect of combining virtual and real to achieve the effect of augmented reality that conforms to the human eye attention mechanism . That is, the present invention first uses the method of saliency detection for an input image to detect the saliency region of the image, and then adjusts the size and placement position of the virtual model based on the detected saliency region , to complete the final augmented reality effect.

As shown in Figure 1, a system for placing virtual objects in augmented reality combined with saliency detection of the present invention includes:

Capture image frame module 1, which is used to read in static images and provide natural and simple scenes. It is the input end of the whole system, and ordinary cameras can be used to capture images. The captured images are images or scenes that need to be augmented reality. The received frame-by-frame data is then transferred to the system for further processing.

The tracking recognition module 2 adopts the simplest, fastest and most effective saliency detection method, and its main task is to carry out saliency detection to the image, and is connected with the acquisition image frame module 1 to collect the images collected by the acquisition image frame module 1. Saliency detection is performed on each frame of image and a saliency map of each image frame can be quickly generated. That is, using the method of image processing to identify the salient area in the input image, and accurately locate the center position and size of the salient area. The salient detection of this module is realized by using the salient detection model of IIIT. IIIT's saliency detection model technology is very mature. Input the image collected by the image frame acquisition module 1 into the saliency detection model of the IIIit, and the model can quickly generate a saliency map of the frame image. The saliency map includes the center position of the salient area, the size and shape of the salient area.

The 3D object generation module 3 mainly imports 3D models for generating virtual objects, which are superimposed on static image frames in the later stage; in the 3D object generation module 3, virtual objects to be superimposed on the real scene are generated, such as when advertising The required 3D virtual mascot model, trademark pattern, or other virtual objects such as fonts. Common software such as 3dmax can be used to generate the 3D model.

The display module 4 is used for displaying virtual-real combined images, and adopts a computer monitor, which is mainly used for displaying image frames superimposed with virtual objects, presenting a virtual-real combined effect.

The image processing module 5 is respectively connected to the acquisition image frame module 1, the tracking recognition module 2, the 3D object generation module 3 and the display module 4, and is used to superimpose the virtual object of augmented reality, and complete the virtual object based on saliency detection in the real scene place. That is, it is used to superimpose a virtual model of production onto a still image. The processing step of described image processing module 5 is,

1) First calculate the ratio of the number of pixels in the salient region detected by the tracking recognition module 2 to the number of pixels in the entire image;

2) Adjust the size of the virtual model generated by the 3D object generation module 3 according to the calculated pixel ratio;

3) Referring to the position of the salient area detected by the tracking and identification module 2, placing the virtual object at the position corresponding to the salient area in the image captured by the image frame acquisition module 1;

4) Send each frame of image where the virtual object is placed to the display module 4 for display.

in,

When the display module 4 displays, if the virtual object is expected to get the most attention of the observer, the image processing module 5 adjusts the virtual object to the same size as the salient area, and superimposes the virtual object on the real scene. The position of the salient area is then sent to the display module 4 for display.

When the display module 4 displays, if it is desired that the virtual object does not affect the main content of the image, the virtual object is placed next to the salient area or at a certain distance from the salient area and then sent to the display module 4 for display.

During the processing of the image processing module 5, when the size or position of the salient region of the image detected by the tracking recognition module 2 changes, the image processing module 5 repeats steps 1) to 4) according to the change of the salient region. The size and position of the object have achieved the effect of real-time combination of virtual and real.

A system for placing virtual objects in augmented reality combined with saliency detection of the present invention can significantly overcome the shortcomings of the two methods for placing virtual objects in the prior art in some application scenarios. The outstanding feature of the present invention is to combine the saliency detection technology to place the virtual object: when the augmented reality system wants to place the virtual object in a real scene or a picture to produce an augmented reality effect, firstly, the saliency detection technology is used , detect the salient area in the image or the scene, and then use the detected salient area as a benchmark to place corresponding virtual objects at different positions relative to the salient area according to the effect that the augmented reality needs to achieve. and adjusting the size of the virtual object according to the size of the salient area. The following examples illustrate the outstanding features of the present invention in some augmented reality application fields.

A very applicable field of application of the present invention is the placement of virtual advertising models in real scenes: assuming that the main content in a scene is a person, and the background part is relatively simple, it is now necessary to add a A virtual 3D model of a brand's mascot to achieve the best advertising effect. The processing steps of the system of the present invention are as follows: first, carry out a salience detection on the scene, and the detected salience area is the area occupied by people in the scene, and then adjust the size of the virtual mascot according to the proportion of the area in the whole scene, Afterwards, the mascot 3D model is placed in the position of the person in the scene, so that the virtual mascot model is in the most prominent position in the scene, that is, the position that can attract people's attention the most, so that the advertising model can be more attractive Moreover, the virtual model adjusts its size according to the change of people in the scene, that is, the change of the size of the salient area, so as to occupy the most prominent position in the scene as much as possible, so as to generate virtual-real interaction. Effect. To sum up, this placement method can achieve the best advertising effect, so that the placed virtual advertising model can attract the most attention of viewers, and at the same time, it can be naturally integrated with the scene to achieve the best advertising effect and virtual-real interactive experience.

In addition to the method of placing the virtual advertising model in the prominent area described above, there are other placement strategies applicable to other situations and requirements: for example, the scene where you want to add the virtual model remains unchanged, and the main content, that is, the prominent area is still a person, But at this time we think that the person in the scene is an important content in the real scene, and we don't want the virtual mascot model to block the person and destroy the content of the real scene. At this time, the salient area of the scene is also detected first, and the detected salient area is the area occupied by people, and then the virtual mascot is placed next to this area or in another position in the non-salient area. In this way, the virtual The model does not affect the main content of the scene. The important content of a scene or a picture is often its salient area. When you do not want the important content to be blocked and you want to place the virtual mascot model, you can use this placement strategy.

In summary, a system for placing virtual objects in augmented reality combined with saliency detection in the present invention fully considers the characteristics of the attention mechanism of human vision, adopts a relatively fast saliency detection method, and automatically adjusts virtual objects And placement, in some augmented reality application fields, such as placing virtual advertising models and other applications, can enable viewers to achieve the best combined virtual and real experience.

Claims (6)

1. A system for placing virtual objects in augmented reality combined with saliency detection, comprising an acquired image frame module (1) for reading in still images, a 3D object generation module (3) for generating virtual objects and a display module (4) for displaying virtual-real combined images, characterized in that it is further provided with:

the tracking identification module (2) is used for carrying out significance detection on each frame of image acquired by the image frame acquisition module (1) and quickly generating a significance image of each frame of image frame;

the image processing module (5) is respectively connected with the image frame acquisition module (1), the tracking identification module (2), the 3D object generation module (3) and the display module (4) and is used for finishing the placement of the virtual object based on the significance detection in the real scene.

2. A system for virtual object placement in augmented reality combined with saliency detection as claimed in claim 1, characterized in that said tracking recognition module (2) performs saliency detection using a saliency detection model of Iiit.

3. A system for virtual object placement in augmented reality with saliency detection as claimed in claim 1, characterized in that the processing steps of said image processing module (5) are,

1) firstly, calculating the proportion of the number of pixels of a salient region detected by a tracking identification module (2) to the number of pixels of the whole image;

2) adjusting the size of the virtual model generated by the 3D object generation module (3) according to the calculated pixel proportion;

3) the position of the salient region detected by the reference tracking identification module (2) is referred, and a virtual object is placed at the position, corresponding to the salient region, of the image acquired by the image frame acquisition module (1);

4) and sending each frame of image with the virtual object to a display module (4) for displaying.

4. A system for placing virtual objects in augmented reality with detection of saliency as claimed in claim 3, characterized in that when the display module (4) is displaying, if the virtual object is expected to get the most attention of the observer, the image processing module (5) adjusts the virtual object to the same size as the saliency region, superimposes the virtual object on the position of the saliency region in the real scene and sends it to the display module (4) for display.

5. A system for placing virtual objects in augmented reality with saliency detection as claimed in claim 3, characterized in that when the display module (4) is displaying, if the virtual object is desired not to affect the main content in the image, the virtual object is placed beside the saliency area or at a distance from the saliency area before being sent to the display module (4) for displaying.

6. The system for placing the virtual object in the augmented reality in combination with the saliency detection as claimed in claim 3, wherein in the processing process of the image processing module (5), when the size or the position of the saliency region of the image detected by the tracking recognition module (2) changes, the image processing module (5) repeats the steps 1) to 4) to dynamically adjust the size and the position of the virtual object according to the change of the saliency region, so as to achieve the effect of real-time virtual-real combination.

Images