KR20200048414A - Selfie support Camera System Using Augmented Reality - Google Patents

Abstract

A selfie support camera system comprises: an augmented reality (AR) terminal which has an embedded imaging camera to photograph an image around a user to acquire real image information, display a three-dimensional virtual image on a display, and display the three-dimensional virtual image corresponding to the real image information and current position information within a view range of the user; and a server to supply the three-dimensional virtual image corresponding to the real image information and the current position information transmitted from the augmented reality terminal to the augmented reality terminal in real time. For a customer intending to use a selfie and a video selfie, a selfie camera recognizes individual user AR stickers when touring a designated area and section, and augmented reality content assigned to each AR sticker is received from the server to automatically generate photographed images.

Description

Selfie support camera system using Augmented Reality

The present invention relates to a camera system, and more particularly, to a self-portrait photography support camera system using augmented reality.

Recently, a portable terminal basically includes a camera, and supports a shooting function using the camera. When the user takes a picture, the user presses a key of the portable terminal or touches a button on the screen to perform camera shooting.

On the other hand, when a user touches a button on the screen when photographing, since the user generally touches with a thumb, the hand held by the portable terminal may shake or the posture of the hand may become unstable when photographing. In particular, self-portrait (self-shooting) ), The portable terminal may shake more.

Therefore, in order to solve this problem, Korean Patent Publication No. 10-2013-0054576, "Method and Apparatus for Self-Self Camera Shooting," has been proposed, but the above-described method and device for self-portrait shooting uses the user's video in the surrounding environment. In addition, it is difficult to automatically shoot by short, medium, and long distance, and it is difficult to provide a service provided by processing and analyzing it.

In addition, when a user performs a self-portrait using his portable terminal, even using a selfie stick, only a narrow range of backgrounds can be photographed.

Recently, content providing using an augmented reality technique that shows a variety of information overlaid on a real space when shooting using a camera module has been actively researched.

Augmented Reality (AR) is a technology belonging to one field of Virtual Reality (VR), and it synthesizes a virtual environment into a real environment that a user feels and makes them feel as if the virtual environment exists in the original real environment. Is a computer technique.

Unlike a conventional virtual reality that targets only virtual spaces and objects, such augmented reality has the advantage of synthesizing virtual objects on the basis of the real world and reinforcing and providing additional information that is difficult to obtain with the real world alone.

That is, the reality implemented by matching a virtual environment made of computer graphics to a live image viewed by a user, for example, a 3D virtual environment, can be referred to as augmented reality. Here, the 3D virtual environment provides necessary information in the live-action image viewed by the user, and the 3D virtual image can be matched with the live-action image to increase the immersion of the user.

Such augmented reality can provide a better sense of reality by providing a photorealistic image in a 3D virtual environment as compared to a simple virtual reality technology, thereby blurring the distinction between the real environment and the virtual environment.

Due to the advantages of the augmented reality system, active research / development of technologies incorporating augmented reality has been actively conducted worldwide. For example, augmented reality technology is in commercialization in the fields of broadcasting, advertising, exhibitions, games, theme parks, military, education and promotion.

Mobile terminals have recently become available for handheld augmented reality systems due to the improvement of computing capabilities of mobile devices such as mobile phones, personal digital assistants (PDAs), and ultra mobile personal computers (UMPCs) and the development of wireless network devices.

As such a system becomes possible, a number of augmented reality applications using mobile devices have been developed. In addition, the spread of mobile devices is becoming very common, and an environment in which augmented reality applications can be accessed is being created.

In addition, user demand for various supplementary services using augmented reality of the terminal is increasing, and attempts to apply various augmented reality contents to the terminal user by using this are increasing.

The present invention is proposed to solve the above technical problem,

Augmented reality where the selfie camera recognizes the individual user's AR sticker and automatically generates the recorded video, including the augmented reality content assigned to each AR sticker when sightseeing in a designated area and section for customers who want to use selfie and self-portrait video. Self-portrait photo support camera system is provided.

In addition, it provides a self-photographed photo support camera system that can accurately identify AR stickers and display virtual objects assigned to real objects at target locations.

According to an embodiment of the present invention, an image camera for capturing real image information by capturing an image around a user is built in and displaying a 3D virtual image on a display, wherein the current location information corresponds to the real image information. An augmented reality terminal displaying a 3D virtual image within a user's field of view, and the 3D virtual image corresponding to the current location information and the real image information transmitted from the augmented reality terminal to the augmented reality terminal in real time The server includes a self-portrait and self-portrait video to the customer who wants to use the self-portrait video. When taking a tour of a designated area and section, the selfie camera recognizes each user's AR sticker and provides AR content assigned to each AR sticker from the server. Self-portrait photo support camera characterized by automatic creation The system is provided.

In addition, the server identifies the real object of the real image information and provides the virtual object of the 3D virtual image allocated to each of the identified real objects to the augmented reality terminal, wherein the augmented reality terminal provides space. Recognize in three dimensions to create a space-based three-dimensional registration coordinate system, and display each virtual object in the coordinates of a real object pre-allocated based on the space-based three-dimensional registration coordinate system. Each time an object is image-identified based on object recognition, the coordinates of the real object are re-recognized, and the identified coordinates of the real object are updated, and a virtual object pre-assigned to the updated coordinates of the real object is displayed. Is done.

In addition, the server selects additional recognition regions by differentiating each actual image information among actual objects existing in the actual image information and having a predetermined image similarity value or higher, and selecting an additional recognition region, and the image difference between the additional recognition regions It is to identify each real object by considering the unique identifier assigned to each real object and the current location information of each real object based on the image difference of the additional recognition area. It is characterized by.

In addition, the server selects additional recognition regions by differentiating each actual image information among actual objects existing in the actual image information and having a predetermined image similarity value or higher, and selecting an additional recognition region, and the image difference between the additional recognition regions It is characterized in that a unique identifier is assigned to each of the real objects based on and a candidate position of the additional recognition area is determined based on the current position information of each real object.

Self-portrait photo support camera system according to an embodiment of the present invention,

When traveling through a designated area and section to a customer who wants to use a selfie or self-portrait video, the selfie camera recognizes each user's AR sticker and automatically generates a captured video including AR content assigned to each AR sticker.

In addition, after accurately identifying the AR sticker, the virtual object allocated to the real object can be displayed at the target position.

By providing AR stickers to customers who want to use self-portraits and self-portrait videos, Selfie Cameras select and classify users when users travel in a designated area and section, and take selfies and videos of users wearing AR stickers by time and place. It can be provided separately for each star and theme.

In addition, a user wearing an AR sticker can be freely exposed to a selfie camera, ignoring a predetermined position and stage, and finally receiving a selfie and a video.

In addition, it is possible to conveniently carry out video shooting in the background of the indoor shooting stage where the user is located, and conveniently transmit video data using the user's personal information.

1 is a view showing a virtual object displayed in a self-portrait photographic support camera system.
2 is a view showing augmented reality implemented on the basis of object recognition
Figure 2a is a view showing augmented reality implemented on the basis of spatial recognition
Figure 3 is a flow chart showing a learning process for identifying similar objects in the self-portrait photographic support camera system
3A is a block diagram showing a process of selecting an additional recognition area for identifying similar objects in a self-portrait photographic support camera system.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings in order to describe in detail that a person skilled in the art to which the present invention pertains can easily implement the technical spirit of the present invention.

1 is a diagram showing a virtual object displayed in a self-portrait photographic support camera system.

Referring to FIG. 1, since a general system displays too many virtual objects, the user's gaze is obscured, and for this reason, it is difficult for a user to recognize necessary information.

In addition, there is a need for a technique capable of accurately identifying and displaying virtual objects assigned to real objects at a target location even if all of the real objects are present or partially present in the captured image.

In addition, if a plurality of real objects having similar appearances are duplicated or unassigned virtual objects are displayed, confusion may be caused. Therefore, a technique for identifying similar objects and preventing duplicate recognition is required.

The self-photographed photo support camera system includes an augmented reality terminal, a server, and a plurality of sensing units. Here, a plurality of sensing units or servers may be selectively provided in the system according to an embodiment.

For reference, if the computing power of the augmented reality terminal and the storage space are sufficient, the system may be configured to perform the role of the server in the augmented reality terminal itself without configuring the server.

In addition, in this embodiment, the augmented reality terminal may be configured as an augmented reality glasses or mobile terminal. The mobile terminal is a generic term for devices that can be used while being carried by a user, such as a mobile phone, a smart phone, and a smart pad. In this embodiment, it will be assumed and described as a mobile terminal composed of a smart phone.

Basically, the augmented reality terminal has a built-in video camera that acquires real image information by capturing the image around the user, and displays a 3D virtual image corresponding to the current location information and real image information in displaying a 3D virtual image on the display. It is displayed within the user's field of view.

The server provides the 3D virtual image corresponding to the current location information and real image information transmitted from the augmented reality terminal to the augmented reality terminal in real time.

The augmented reality terminal is basically configured to provide satellite location information to the server as current location information. When the communication module is included in the augmented reality terminal, the location of a nearby Wi-Fi repeater and the location of a base station, as well as satellite location information, may be additionally provided to the server as current location information.

For example, since satellite location information is often not received indoors, the augmented reality terminal may further detect and transmit signal strengths of at least one searched Wi-Fi repeater to a server. That is, since the absolute location of the Wi-Fi repeater located in the room is stored in advance on the server, if the augmented reality terminal additionally provides the unique number and signal strength of the searched Wi-Fi repeater, the server augments It is possible to grasp the relative movement path of the real terminal.

That is, the relative distance between the augmented reality terminal and the Wi-Fi repeater can be confirmed as the signal strength, and the moving direction can be calculated based on the change in the signal strength with the neighboring Wi-Fi repeater.

Therefore, the server identifies the virtual objects assigned to each real object through the current location information of the user who wears or possesses the augmented reality terminal and the actual image information captured by the video camera of the augmented reality terminal, and augments the information about it. Real-time transmission to a real terminal.

On the other hand, when the customer wants to use the selfie and self-portrait video, the selfie camera recognizes each user's AR sticker and automatically receives the augmented reality content assigned to each AR sticker from the server when sightseeing the designated areas and sections. Can be created.

2 is a view showing augmented reality implemented based on object recognition, and FIG. 2a is a view showing augmented reality implemented based on spatial recognition.

Referring to FIGS. 2 and 2A, first, a method of identifying a real object based on object recognition is easy because image size is small compared to space. However, recognition is possible only when the object is included in the image, and limitations may occur in the recognition direction and distance. It is a difficult task to store a location in advance because things can change location. For reference, one object-based three-dimensional matching coordinate system (augmented reality matching coordinate system) is generated for each object.

Next, the method of identifying the location of the real object based on spatial recognition basically applies the technology of Simulaneous Localization And Map-Building, Simultaneous Localization and Mapping (SLAM). Slam (SLAM) is defined as a core technology for autonomous driving by creating an accurate three-dimensional map of the environment without external assistance with only sensors attached to the terminal (robot) while the terminal (robot) moves around in an unknown environment. Can be.

That is, the spatial recognition-based method is defined as a method of recognizing space in three dimensions to generate a three-dimensional coordinate system of space, and displaying each virtual object in the coordinates of a real object previously allocated based on the three-dimensional coordinate system. Can be.

Therefore, even if the object is not included in the image, the virtual object can be augmented with the object (real object). However, when the terminal is moved from the initial recognition point, the spatial tracking error may accumulate and it may be difficult for the augmented reality information (virtual object) to be displayed at an appropriate location. For reference, one space-based three-dimensional registration coordinate system (augmented reality matching coordinate system) is generated per space.

Therefore, the system of the present invention can accurately identify and display virtual objects assigned to real objects at a target location even if all or all of them exist in the captured image or partially exist. It is configured to apply the object recognition at the same time to display the location of the virtual object at the correct location.

That is, the augmented reality terminal recognizes the space in three dimensions to generate a space-based three-dimensional matched coordinate system and displays each virtual object in the coordinates of a real object previously allocated based on the space-based three-dimensional matched coordinate system. However, whenever a real object of real image information is image-identified based on object recognition, the coordinates of the real object are grasped, and then a 3D match coordinate system based on object recognition is additionally generated, based on a 3D match coordinate system based on object recognition. Update the space-based 3D registration coordinate system.

3 is a flow chart showing a learning process for identifying similar objects in the self-photographed photo support camera system, and FIG. 3A is a configuration showing a process of selecting an additional recognition area for identifying similar objects in the self-photographed photo support camera system It is.

3 and 3A, the server operates to identify real objects of real image information and to provide augmented reality terminals with virtual objects of 3D virtual images respectively assigned to the identified real objects.

That is, the server selects an additional recognition region by differentiating each actual image information among real objects having a predetermined image similarity value (d) or more among a plurality of real objects present in the real image information, and the difference in the images of the additional recognition regions On the basis of, a unique identifier is assigned to each real object.

For example, if the additional recognition area has different characters or numbers, the server assigns a unique identifier to each real object based on the difference in the additional recognition area, stores it in a database, and stores the virtual identifier assigned to the unique identifier. The object may be transmitted to an additional augmented reality terminal.

That is, when a plurality of real objects have a predetermined image similarity value (d) or higher, abstract the image and differentiate the image to set an additional recognition area (additional learning area), and then identify differences in the additional recognition area It is to give each unique identifier.

Through this method, it is possible to identify each AR sticker possessed by the user.

The self-photographed photo support camera system is configured to exchange data with each other using a communication environment or communication system called M2M (machine-to-machine) communication or Internet of Things (IoT).

Self-photographed photo support camera system, if a user attaches an AR (augmented reality, AR) sticker with a specific QR code or mark, the image capture unit recognizes the code or mark, and then permits it according to the recognized value Recognized as a person, the user wearing the AR sticker can be freely taken by the selfie camera of the image capturing unit ignoring a predetermined position and stage.

In other words, by providing AR stickers to customers who want to use self-portraits and self-portrait videos, the selfie camera selects and sorts users when they travel the designated areas and sections, and separates selfies and videos of users wearing AR stickers by time. , It can be provided separately by place and theme.

In addition, a user wearing an AR sticker can be freely exposed to a selfie camera, ignoring a predetermined position and stage, and finally receiving a selfie and a video.

The sensing units are respectively disposed on a plurality of indoor shooting stages (STAGE), and receive personal information from a user's portable terminal using an RF tagging method. In addition, the sensing unit is configured to allow payment for photography. The sensing unit 100 is defined as a fixed terminal that transmits location information so that a user can perform selfie in the background of the current indoor shooting stage STAGE.

That is, in this embodiment, the indoor shooting stage (STAGE) refers to a designated photographing location so as to take photos against an exhibition, fair, or theme space, and a sensing unit is installed in each indoor shooting stage (STAGE). For example, in an exhibition or exhibition where a plurality of booths are installed, each booth may be defined as one indoor photography stage (STAGE).

A pressure sensor is disposed on the bottom surface of the plurality of indoor shooting stages STAGE, and the sensing unit may detect the user's location based on the detection result of the pressure sensor and transmit the information to the image capturing unit. Therefore, the image capturing unit may proceed to take the image centering on the location where the user is located.

When a user inputs at least one personal information of a phone number, an email address, and a social network service (SNS) account into the sensing unit and performs payment, the sensing unit controls the video recording unit to shoot with a preset background It supports users to take self-portrait photos while displaying the scheduled video in real time.

At this time, the image captured by the image capturing unit is transmitted to the data server, and the data server transmits the captured image using the input personal information. That is, the data server transmits a photograph taken using a phone number, an email address, and a social network service account.

The image capturing unit is installed at a position where a picture can be taken against the background of each indoor shooting stage (STAGE).

For example, a plurality of image photographing units may be installed on the ceiling portion of the exhibition hall building, and each image photographing unit is configured to be in charge of a plurality of indoor shooting stages STAGE. At this time, the image capturing unit may operate in conjunction with the lighting installed on the ceiling. That is, after determining the brightness of the photographing area, the image photographing unit may automatically adjust the brightness of the photographing area by controlling the brightness of the light installed on the ceiling portion and the illumination angle.

In addition, when it is determined that there is an obstacle in the photographing area through the image object chain, the image photographing unit may automatically move the user's body to be photographed. That is, when the image capturing unit is installed on the ceiling of the exhibition hall building, it is preferable that a plurality of image capturing units are provided with movable rails that can be automatically moved, and the image capturing unit is equipped with a movable driving unit to move to a predetermined position along the movable rail. Can be.

The shooting direction and the zoom magnification of the image capturing unit are controlled according to the control of the sensing unit, and these scheduled images are displayed on the display unit or portable terminal of the sensing unit in real time, so that the user can easily take a self-portrait picture.

That is, the image photographing unit photographs an image in the background of any one of the indoor photographing stages STAGE selected from a plurality of indoor photographing stages STAGE, and photographs the image based on the location information transmitted from the sensing unit and the user's personal information. .

Here, since the image includes both a moving image and a still image, the image capturing unit may selectively take the moving image and the still image (photo) at the user's option.

According to an embodiment, one image photographing unit may be configured to be in charge of one indoor photographing stage STAGE, and by adjusting the photographing direction and the zoom magnification, one image photographing unit may include a plurality of pre-allocated indoor photographing stages STAGE. It may be configured to handle each.

A PTZ camera may be used for video shooting, and the PTZ camera is a camera capable of adjusting left / right / up / down rotation (PAN), tilt (TILT), and zoom (ZOOM) using a motor. Therefore, the PTZ camera can be rotated in a specific angle and direction under the control of the sensing unit, so that it is possible to adjust the area where the image can be taken.

The image capture unit is equipped with an emotional LED (LED) for nighttime identification, and an auxiliary floodlight is installed to enable self-photography at night, but the search headlight is illuminated by the indoor shooting stage (STAGE) to facilitate image acquisition.

The video recording unit basically selects a camera with a mega-pixel or higher resolution, and divides it into near, medium, and long-distance types, and the camera uses an IP network camera or a DSLR digital camera (zoom lens-mounted type). It is operated by dividing it into a method of driving a zoom lens that automatically takes a picture of each picture by distance, and a method of acquiring a high-resolution picture of the entire background and automatically increasing the proportion of the picture to display pictures by distance.

Also, the image capturing unit may transmit data using a wireless mesh network method.

That is, when a plurality of image photographing units are provided, each image photographing unit wirelessly relays data to each other, thereby extending the data exchange limit distance between the sensing unit, the portable terminal, and the data server.

For reference, a microphone is disposed at a 90-degree angle difference in the image capturing unit so that a total of four microphones can be configured to collect sound in each direction. At this time, a partition wall is installed between the four microphones to assist in determining the source of the sound. It is preferable that a directional microphone is used as the microphone. When a directional microphone is provided as described above, the image capturing unit may detect a user's voice and detect a moving direction in advance.

The data server stores the image data transmitted from the image capturing unit and the user's personal information as a database, and uses the personal information such as a phone number, email address, and social network service account previously input by the user to send the image data to the user. send. The image data may be produced in the form of an electronic album and transmitted to a user.

Therefore, the user can take a self-portrait at each indoor shooting stage (STAGE), and receive a video captured by a phone number, email address, and social network service account that he / she has previously input.

Meanwhile, the sensing unit may receive the user's personal information, such as a telephone number, an email address, and a social network service account, through a wireless communication method from the user's portable terminal.

At this time, since the sensing unit and the portable terminal are configured to exchange data through a wireless communication method such as Bluetooth or NFC (Near Field Communication), the user simply transmits personal information through a method of tagging the portable terminal to the sensing unit. Can be.

In addition, when the user tags the portable terminal in the sensing unit, a link to download a photo shooting application is displayed.

When a user installs a photographing application on a portable terminal, the user can control the image photographing unit for a predetermined time using a photographing application installed on the portable terminal as well as the sensing unit.

That is, the portable terminal can display a scheduled image in real time, and transmit a control signal instructing to proceed with the photographing at a specific point in time. In addition, it may be configured to pay an amount for taking an image through a photo shooting application of the portable terminal.

As described above, when the sensing unit receives the user's personal information through the wireless communication method from the user's portable terminal, the sensing unit or the image capturing unit may be configured to transmit the scheduled image to the portable terminal in real time.

You can request additional photos of the past. That is, when a user requests a picture taken in the past at the current place through the portable terminal, the corresponding picture may be displayed on the portable terminal. In addition, it is possible to obtain a desired picture by driving a self-portrait picture, which is difficult to obtain by manipulating it with one's own hands, in real-time by operating a video recording unit pre-installed at a short distance, medium distance, or long distance.

Self-portrait photo support camera system according to an embodiment of the present invention,

When traveling through a designated area and section to a customer who wants to use a selfie or self-portrait video, the selfie camera recognizes each user's AR sticker and automatically generates a captured video including AR content assigned to each AR sticker.

In addition, after accurately identifying the AR sticker, the virtual object allocated to the real object can be displayed at the target position.

As such, those skilled in the art to which the present invention pertains will appreciate that the present invention may be implemented in other specific forms without changing the essential features of the technical concept. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims rather than the above detailed description, and it should be interpreted that all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof are included in the scope of the present invention. do.

Claims (4)

An image camera that captures images around the user to obtain real image information is built in, and when displaying a 3D virtual image on a display, the user's field of view of the 3D virtual image corresponding to the current location information and the real image information is displayed. Augmented reality terminal displayed within; And
It includes; a server that provides the 3D virtual image corresponding to the current location information and the real image information transmitted from the augmented reality terminal to the augmented reality terminal in real time;
Self-portraits and self-portraits are used by self-portrait cameras. When taking a tour of a designated area and section, the selfie camera recognizes the individual user's AR stickers and automatically generates augmented video content by receiving the augmented reality content assigned to each AR sticker from the server. Self-portrait photo support camera system, characterized in that.
According to claim 1,
In the server, the real object of the real image information is image-identified, and the virtual object of the 3D virtual image allocated to each of the identified real objects is provided to the augmented reality terminal.
The augmented reality terminal recognizes space in three dimensions to generate a space-based three-dimensional matched coordinate system and displays each virtual object in the coordinates of a real object previously allocated based on the space-based three-dimensional matched coordinate system. However, whenever the real object of the real image information is image-identified based on object recognition, the coordinates of the real object are re-recognized, and then the coordinates of the identified real object are updated and allocated to the updated coordinates of the real object in advance. Self-portrait photo support camera system, characterized by displaying the virtual object.
According to claim 2,
The server,
Among the plurality of real objects present in the real image information, real objects having a predetermined image similarity value or higher select different recognition regions by differentiating the respective real image information, and actualize a unique identifier based on the image difference of the additional recognition regions. To each object,
Self-photographed photo support camera system characterized by identifying each real object in consideration of both the unique identifier assigned to each real object and the current location information of each real object based on the difference in the image of the additional recognition area.
According to claim 2,
The server,
Among the plurality of real objects present in the real image information, real objects having a predetermined image similarity value or higher select different recognition regions by differentiating the respective real image information, and actualize a unique identifier based on the image difference of the additional recognition regions. To each object,
Self-photographed photo support camera system, characterized in that the candidate position of the additional recognition area is determined based on the current position information of each real object.

Images