KR101923177B1 - Appratus and method for providing augmented reality information based on user - Google Patents

Abstract

The present invention implements a method of providing user-based augmented reality (AR) information. To this end, the present invention drives two cameras mounted in opposite directions to each other in a mobile terminal at the same time, acquires user information by recognizing a face from a user's face image through a front camera, and acquires user information from at least one Determining augmented reality information for the object based on the user information, and superimposing the determined augmented reality information on the object and displaying the augmented reality information. At this time, when the object moves, a feature point of the object is extracted, the feature point is tracked, the coordinates for superimposing the augmented reality information on the tracked object are calculated, and the augmented reality information is arranged Is added. By doing so, the augmented reality service can be differentiated and provided to the user according to the information of the user.

Description

[0001] APPARATUS AND METHOD FOR PROVIDING AUGMENTED REALITY INFORMATION BASED ON USER [0002]

The present invention relates to a mobile terminal and a method for providing augmented reality information using the same, and more particularly, to an apparatus and method for providing user-based augmented reality information using a camera function installed in a mobile terminal.

2. Description of the Related Art Recently, various services and additional functions provided by a mobile terminal have been gradually expanded. In order to increase the utility value of the mobile terminal and satisfy various needs of the users, it is necessary to utilize the mobile terminal having the camera and various services and additional functions of the mobile terminal. As an example of such utilization, there is a method of displaying various information about an object in a camera image to a user, and an Augmented Reality (AR) technique can be applied to this.

Generally, augmented reality means combining virtual objects that the user wants to express based on the real world. In other words, it combines a virtual object into a real environment and expresses it as if it exists in the original environment. Since the virtual object needs to be increased to the correct position without any error even if the position of the user's camera or the pose of the user changes or the virtual object is moved, the real-time image recognition technology is widely used.

Conventional augmented reality services compare the camera image (or processed information) currently captured and the previously stored database image (or processed DB information) by using the marker and markerless recognition method, It is based on showing the reinforcement on the above.

This augmented reality method has a limit in that virtual object information prepared by the service provider is equally provided to all users who are provided with the service. However, if different virtual object information can be provided for each user, various enhancement information differentiated from the user can be provided. For example, if a movie poster is recognized and a movie trailer is matched to a poster on a poster, if the user knows information about the language used, it is possible to display subtitles in the language used by the user.

Thus, it is required to improve the software part of the mobile terminal in order to diversify the camera utilization plan of the mobile terminal.

Accordingly, the present invention provides an apparatus and method for providing user-based augmented reality information using a camera function installed in a mobile terminal.

According to another aspect of the present invention, there is provided an apparatus for providing user-based augmented reality information, comprising: a first image input unit for outputting a first image of a user's face; A second image input unit for outputting a captured second image; a display unit for displaying the second image; and a display unit for recognizing the user face from the first image and acquiring user information matched with a result of recognizing the user face A second image processing unit for extracting at least one object in the second image and acquiring augmented reality information for the object corresponding to the user information; And a controller for superimposing the augmented reality information and displaying the augmented reality information on the display unit.

According to another aspect of the present invention, there is provided a method for providing user-based augmented reality information in a mobile terminal having a first image input unit and a second image input unit, The method includes the steps of simultaneously driving two image input units, receiving a first image of a user's face through the first image input unit, recognizing the user face from the first image, Acquiring user information matched to the first image input unit, receiving a second image captured in a direction opposite to the first image input unit through the second image input unit, extracting at least one object in the second image, Acquiring augmented reality information for the object corresponding to the user information; And displaying the real information in a superimposed manner.

According to the present invention, there is an advantage that quality and satisfaction of a service can be improved by differentiating and providing an augmented reality service to a user according to information of a user. In addition, the present invention can provide the content of the contents in real time in accordance with the user even if the user changes during the augmented reality service. In addition, the present invention can be extended to various augmented reality service scenarios based on various recognition technologies and hardware configurations.

1 is a front view and a rear view of a mobile terminal according to an embodiment of the present invention,
2 is an internal block diagram of a mobile terminal according to an embodiment of the present invention;
3 is a flowchart illustrating an operation of a mobile terminal for providing user-based augmented reality information according to an exemplary embodiment of the present invention.
FIG. 4 is a flowchart illustrating an operation of a mobile terminal for providing user-based augmented reality information according to another embodiment of the present invention;
FIG. 5 is a diagram illustrating a screen in which the augmented reality information for an object in a camera image is superimposed on the augmented reality information according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It will be appreciated that those skilled in the art will readily observe that certain changes in form and detail may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims. To those of ordinary skill in the art.

The present invention implements a method of providing user-based augmented reality (AR) information. To this end, the present invention drives two cameras mounted in opposite directions to each other in a mobile terminal at the same time, acquires user information by recognizing a face from a user's face image through a front camera, and acquires user information from at least one Determining augmented reality information for the object based on the user information, and superimposing the determined augmented reality information on the object and displaying the augmented reality information. At this time, when the object moves, a feature point of the object is extracted, the feature point is tracked, the coordinates for superimposing the augmented reality information on the tracked object are calculated, and the augmented reality information is arranged Is added. By doing so, the augmented reality service can be differentiated and provided to the user according to the information of the user.

The case where the camera is mounted in the mobile terminal used in the present invention is as shown in FIG. 1 is a perspective view of a mobile terminal 100. FIG. Referring to FIG. 1, a first image input unit 105 may be mounted on a front surface of the mobile terminal 100, and a second image input unit 110 may further be mounted on a rear surface thereof.

The first image input unit 105 and the second image input unit 110 have opposite shooting directions. Since the first image input unit 105 faces the user, the second image input unit 110 takes a role of shooting a user's face at the time of self-photographing, and the second image input unit 110 is directed toward the outside.

In the above description, the second image input unit 110 is disposed on the rear side of the mobile terminal 100, but the present invention is not limited thereto.

Hereinafter, the operation of the mobile terminal and the components thereof will be described with reference to FIG. The mobile terminal of the present invention is not limited to the mobile phone as shown in FIG. 1, but may be applied to any type of imaging apparatus capable of image capturing and capable of performing face recognition of a subject. For example, the mobile terminal may be a device supporting a photographing function such as a smart phone, a mobile phone, a tablet PC, a digital camera, and the like.

Referring to FIG. 2, the mobile terminal 100 mainly includes a part 205 for recognizing a first image, a part 215 for superimposing and reproducing augmented reality information on an object in a second image, An output unit 217, a storage unit 219, and a control unit 220.

The portion 205 related to the first image includes a first image input unit 105 and a first image processing unit 200 for processing a first image photographed through the first image input unit 105, The portion 215 related to the second image includes a second image input unit 110 and a second image processing unit 210 for processing a second image photographed through the second image input unit 110. The first image processor 200 and the second image processor 210 may be included in the control unit 220. The first image processor 200 and the second image processor 210 may be included in the controller 220, The control unit 220 may take charge of the function of the control unit.

First, the first image input unit 105 means a camera, and captures an image or a moving image for a user. The image signal of the user is transmitted to the first image processing unit 200. The first image processing unit 200 processes the image signal in units of frames and performs face recognition on the photographed image. Here, the face recognition can be applied not only to still images such as photographing but also to moving images. For example, since moving images consist of continuous still image frames, still image frames can be used for face recognition.

At this time, the face recognition is performed based on the face recognition algorithm provided in the mobile terminal 100, and a method of identifying using the entire outline information of the face of the subject is used, or each component included in the face, , Mouth, and so on. And a step of detecting the position of the face in the image by a preprocessing process for recognizing the face.

The first image processing unit 200 may include a face region detecting unit 225, a face feature extracting unit 230, a face recognizing unit 235, and a recognition result matching unit 240.

Specifically, when the face region detection unit 225 detects a face region of the subject from the photographed image, the face feature extraction unit 230 extracts minutiae points of the entire eye, eyebrows, nose, lips, etc. from the detected face region, Feature information usable for recognition is extracted from the outline. By doing so, the face recognition unit 235 can identify the eyes, nose, mouth, and the like of the subject or acquire the feature information that can be used for recognition from the subject face. Alternatively, the face recognition unit 235 compares the photographed image with a face image stored in the storage unit 219 in advance to perform face recognition. The pre-stored facial image includes facial feature information or facial components such as eyes, nose, mouth, and the like.

When the face recognition is completed through this method, the recognition result matching unit 240 determines the user information matching the face recognition result. The recognition result matching unit 240 determines user information such as the name, age, sex, nationality, etc. of the subject using the mobile terminal 100 from the face recognition result based on the information stored in the storage unit 219. [

 In this manner, the first image processing unit 200 performs recognition of the image of the user based on the face recognition algorithm, and extracts user information based on the recognition result.

On the other hand, the second image input unit 110 photographs in a direction opposite to the first image input unit 105. For example, when a user is illuminated by a camera corresponding to the second image input unit 110 of the mobile terminal, the image of a nearby object is displayed as a preview image. To this end, the photographed image from the second image input unit 110 is transmitted to the second image processing unit 210. The second image processing unit 210 extracts and recognizes at least one object in the photographed image, A detailed description, a virtual object, and the like can be displayed as a stereoscopic image. At this time, detailed descriptions, virtual objects, and the like superimposed on the object are based on user information.

The video output unit 217 displays various types of menus of the mobile terminal 100 as well as information input by the user or information provided to the user. Particularly, the image output unit 217 can display a photographed image. The image output unit 217 may display and set a part of the display area as a face image area photographed through the first image input unit 105 to enable self photographing. At this time, it is preferable that the face image area for self photographing is displayed only during self photographing. Alternatively, the image output unit 217 may display only images captured through the second image input unit 110. In this case, a separate accessory, such as a mirror, may be mounted on the mobile terminal 100 for photographing a face image to be used for face recognition so as to facilitate self-photographing.

The storage unit 219 stores the operating system of the mobile terminal 100, various applications, information input to the mobile terminal 100, and information generated therein. For example, the storage unit 219 stores images such as photographs and moving images, and stores a face recognition algorithm for recognizing a subject face input through the first image input unit 105 and data used for face recognition do. For example, information on a face to be used for face detection and face characteristic information per user registered in advance for use in face recognition can be stored.

The storage unit 219 stores the augmented reality information for each object required to provide the augmented reality service. At this time, the augmented reality information may be provided in real time based on the current location through an external communication network. For example, when the mobile terminal illuminates the surroundings with the camera, details such as detailed descriptions of the nearby objects are displayed as stereoscopic images.

In addition, the storage unit 219 stores algorithms necessary for detecting and tracking moving objects in an image input in real time through the second image input unit 110. [ For example, the storage unit 219 stores feature points and the like for a target object to be superimposed on the augmented reality information through image matching.

The second image processing unit 210 includes a feature point extracting unit 245, a feature point matching unit 250, a feature point tracking unit 255, a coordinate calculation unit 260, And a rendering unit 265.

In this case, when the object in the real world in which the photographing position, pose, etc. of the camera changes or the augmented reality information is overlapped, it is important to enhance the augmented reality information to an accurate object position without error. For this purpose, it is desirable to extract the feature information from the portion where the change of the image pattern is conspicuous while minimizing the feature information of the input image when performing object detection and tracking in the captured image. Herein, the feature point refers to a representative pixel or a set of pixels to be compared in each image frame, and may be a corner edge which is mainly detected in an image frame, a border having a large fluctuation width such as a contrast or color, etc. .

Accordingly, the feature point extracting unit 245 extracts feature information of the input image through the second image input unit 110, that is, feature points. For this purpose, general feature point extraction and matching methods can be used as is or modified appropriately. For example, a characteristic map representing a spatial change of an image pattern in a frame in an input image is generated to extract a feature point of the image, or an ROI is determined based on the position of an arbitrary reference point in the input image, A method of extracting feature points while updating a region of interest in a next image according to a feature point matching and tracking result in the next step, and the like.

The feature point matching unit 250 matches the extracted feature points with the feature points stored in the storage unit 219, and the feature point tracking unit 255 tracks the matching feature points. For example, when minutia information about at least one object is stored in advance in the storage unit 219, the minutia matching unit 250 compares the minutia information about the previously stored object with the minutia information about the object in the input image Find matching points. Thus, even if the photographing position of the camera changes or the subject in the input image moves, the position, size, pose, and the like of the object in the input image can be tracked.

The coordinate calculation unit 260 calculates coordinates of a camera to increase the augmented reality information for the object by calculating the motion of the camera.

The rendering unit 265 creates a two-dimensional image in which the augmented reality information for the object is placed in the calculated coordinates in the input image.

The control unit 220 controls the overall operation of the mobile terminal 100. The control unit 220 according to the embodiment of the present invention performs user recognition from the facial image through the first image input unit 105 to determine user information. At the same time, the control unit 220 controls the second image input unit 110 to perform object detection and tracking in an image input in real time. Thus, even if the object moves in the input image, the rendering is controlled so that the augmented reality information for the object is superimposed on the position corresponding to the movement. That is, the controller 220 tracks the position change of the object and controls the augmented reality information to be superimposed on the tracked position.

In particular, the control unit 220 determines information to be rendered by the rendering unit 265 and contents to be output by the video output unit 217 in accordance with the user information. Specifically, the controller 220 determines the augmented reality information for the object based on the user information such as the name, age, sex, and nationality of the object determined by the recognition result matching unit 240, So as to be displayed.

As described above, according to the present invention, it is possible to improve the limitation of using only one camera, such as photographing an external subject using a camera mounted on the rear surface of a mobile terminal, or taking a self- And can increase the utility of the camera. Furthermore, it is possible to provide various differentiated augmented reality information according to the result of face recognition, thereby satisfying various users' tastes.

Hereinafter, the operation of the mobile terminal 100 having the above configuration will be described with reference to FIGS. 3 and 4. FIG. FIG. 3 illustrates a case where a camera mounted on the mobile terminal 100 is simultaneously driven according to an embodiment of the present invention. FIG. 4 illustrates a case where a camera is sequentially driven according to another embodiment of the present invention. .

Referring to FIG. 3, when the augmented reality service application is executed, the mobile terminal 100 may enter a mode for simultaneously operating the first image input unit 105 and the second image input unit 110 in operation 300. This mode is roughly divided into a first process 305 for processing a first image in the first image input unit 105 and a second process 310 for processing a second image in the second image input unit 110 . The first step 305 is a step of recognizing a user who uses the mobile terminal 100 through a first image input through the first image input unit 105, Is a process of tracking the object in a second image input through the second image input unit 110, for example, an image including a background and an object.

First, when the face image is input through the first image input unit 105, the mobile terminal 100 performs recognition of the face image input in operation 306. In operation S305, For this, a face recognition algorithm can be used. When the face recognition result is output, the mobile terminal 100 searches for user information matching the recognition result in step 307. [ Accordingly, the mobile terminal 100 can know who is currently using the mobile terminal 100. For example, the mobile terminal 100 can acquire user information such as a user's name, age, gender, nationality, etc. through the search of the user information, or otherwise determine which user is registered in advance have.

In operation 310, when the mobile terminal 100 receives a surrounding image including at least one object through the second image input unit 110, the mobile terminal 100 extracts feature points . Then, it is determined whether there is a change in the position, size, pose, or the like of the object through matching of the minutiae extracted in step 312 and the minutia information of the object stored in advance. If there is no change, the process proceeds to step 315, and if there is a change in the position, size, pose, or the like of the object, the process proceeds to step 313 to update the position, size, and pose of the object. In this way, the position, size, pose, and the like of the object are updated to track the change of the object according to the movement of the object. In order to track the object, a method of comparing the position of the feature point extracted from the previous image frame with the position of the feature point extracted from the current image frame may be used.

Then, the mobile terminal 100 determines the augmented reality information for the object based on the user information in step 315, and displays the augmented reality information on the object in step 320 in step 320. By doing so, the augmented reality service can be configured and provided according to the user information.

Meanwhile, a case in which cameras are sequentially driven according to another embodiment of the present invention will be described with reference to FIG.

Referring to FIG. 4, when the augmented reality service application is executed, the mobile terminal 100 first drives the first image input unit 105 in step 400. Referring to FIG. In step 405, the facial image input through the first image input unit 105 is recognized. In step 410, user information matching with the recognition result is retrieved. The operations in steps 405 and 410 are the same as those in steps 306 and 307 of FIG.

However, in another embodiment of the present invention, there is a difference in that the second image input unit 110 is driven in step 415 after the first image input unit 105 is driven. Therefore, the operations in steps 420 to 445 are the same as those in steps 311 to 320 of FIG. 3, so a detailed description thereof will be omitted.

Meanwhile, FIG. 5 illustrates a screen in which the augmented reality information for the object in the camera image is superimposed on the augmented reality information according to the embodiment of the present invention. For example, when a user of the mobile terminal 100 visiting a museum photographs a periphery where the art work is hanged, a preview screen on which a surrounding image including the art work is displayed can be seen.

A peripheral image photographed through the second image input unit 110 is displayed on the screen 500 of the mobile terminal 100. Augmented reality information based on user information may be displayed together with the peripheral image. For example, the art objects 510 and 515 correspond to the objects included in the surrounding image, and the detailed descriptions 520, such as the art name, the artist, and the description of the art, as the augmented reality information for the art works 510 and 515, 525 may be superimposed and displayed. The augmented reality information may be previously stored in the storage unit 219 of the mobile terminal 100, or may be acquired from an external server. For example, when the mobile terminal 100 enters the art gallery, the augmented reality information can be downloaded from the server in the art gallery.

At this time, if the user's face is photographed through the camera toward the user, the detailed description can be displayed in Korean if the user's nationality is Korean. If the user's nationality is in English, the detailed description will be displayed in English. As described above, the Augmented Reality technique is applied when superimposing detailed descriptions on the surrounding images. Therefore, when the position of the object in the surrounding image is changed or the photographing direction is changed, the position at which the augmented reality information is displayed is changed so as to correspond to the changed image.

Meanwhile, FIG. 5 illustrates a case where the augmented reality information superimposed and displayed on an object in the surrounding image is a detailed description, but it may be implemented as a moving image, a virtual object, or the like. For example, when a user shoots a movie poster, the mobile terminal 100 may extract feature points included in the movie poster, and then display a movie trailer on the movie poster. At this time, the movie trailer may be a trailer having a view value suitable for the user's age range based on the result of the user's face recognition. Since the present invention is based on the information of the user, it is possible to display the augmented reality information suitable for the user who currently has the mobile terminal 100 together with the surrounding images.

It will be appreciated that embodiments of the present invention may be implemented in hardware, software, or a combination of hardware and software. Such arbitrary software may be stored in a memory such as, for example, a volatile or non-volatile storage device such as a storage device such as ROM or the like, or a memory such as a RAM, a memory chip, a device or an integrated circuit, , Or stored in a machine-readable storage medium, such as optical, or magnetically recordable, such as, for example, a CD, DVD, magnetic disk or magnetic tape. It will be appreciated that the storage is an example of a machine-readable storage medium suitable for storing programs or programs containing instructions for implementing the embodiments of the present invention. Accordingly, the present invention includes a program including code for implementing the method recited in any claim of this specification, and a machine-readable storage medium storing such a program. In addition, such a program may be electronically transported through any medium such as a communication signal transmitted via a wired or wireless connection, and the present invention appropriately includes the same.

Also, the mobile terminal can receive and store the program from a program providing apparatus connected by wire or wireless. The program providing apparatus includes a memory for storing a program including instructions for the mobile terminal to perform a method for providing an augmented reality service based on a user information, information required for a method for providing the augmented reality service, A communication unit for performing wired or wireless communication with the mobile terminal, and a controller for requesting the mobile terminal or automatically transmitting the program to the mobile terminal.

Claims (15)

An apparatus for providing user-based augmented reality information,
A first image input unit for outputting a first image of a user's face,
A second image input unit for outputting a second image photographed in a direction opposite to the first image input unit;
A display unit for displaying the second image;
A first image processor for recognizing the user face from the first image and acquiring user identification information matched with a result of recognizing the user face;
A second image processing unit for extracting at least one object in the second image and acquiring augmented reality information for the object corresponding to the user identification information;
And a controller for superimposing the augmented reality information on the object in the second image and displaying it on the display unit.
The apparatus of claim 1,
When the augmented reality service application is executed, the first image input unit and the second image input unit are simultaneously driven.
The apparatus of claim 1, wherein the first image processor comprises:
A face region detecting unit for detecting a user face region from the first image;
A face feature extraction unit for extracting a face feature from the detected face region,
A face recognition unit for recognizing the user face using the extracted face feature,
And a recognition result matching unit for searching for user identification information corresponding to a result of recognizing the user's face.
The image processing apparatus according to claim 1,
A feature point extracting unit that extracts feature points of at least one object in the second image,
And a feature point tracking unit for tracking the object using the extracted feature points.
5. The method of claim 4,
And a feature point matching unit for matching the extracted feature points with previously stored feature points.
6. The apparatus according to claim 5,
And tracking the object by tracking the matched minutiae points. ≪ Desc / Clms Page number 19 >
5. The apparatus of claim 4,
And controls the positional change of the object to be tracked so that the augmented reality information is superimposed and displayed at the tracked position.
2. The method according to claim 1,
And at least one of age, name, sex, and nationality of the user.
The method as claimed in claim 8,
Wherein the user-based augmented reality information is displayed in a language corresponding to the user's nationality.
A method for providing user-based augmented reality information in a mobile terminal having a first image input unit and a second image input unit,
When the augmented reality service application is executed, simultaneously operating the first image input unit and the second image input unit;
Receiving a first image of a user's face through the first image input unit;
Recognizing the user face from the first image and acquiring user identification information matching the recognition result of the user face;
Receiving a second image photographed in a direction opposite to the first image input unit through the second image input unit;
Extracting at least one object in the second image and acquiring augmented reality information for the object corresponding to the user identification information;
And a step of superimposing the augmented reality information on the object in the second image and displaying the augmented reality information.
The method of claim 10, wherein the step of acquiring user identification information comprises:
Detecting a user's face region from the first image;
Extracting a face feature from the detected face region;
Recognizing the user face using the extracted face feature;
And searching for user identification information corresponding to a result of recognizing the user's face.
12. The method according to claim 11,
Wherein the information includes at least one of age, name, sex, and nationality of the user.
13. The method according to claim 12,
Wherein the user-based augmented reality information is displayed in a language corresponding to the user's nationality.
The method of claim 10, wherein the extracting of at least one object in the second image comprises:
Extracting feature points for at least one object in the second image;
And tracking the object by using the extracted minutiae points.
15. The method of claim 14,
And tracking the change of the position of the object and displaying the augmented reality information in a superimposed manner at the tracked position.

Images