CN111311759B - Mobile augmented reality video editing system - Google Patents
Mobile augmented reality video editing system Download PDFInfo
- Publication number
- CN111311759B CN111311759B CN201910071215.5A CN201910071215A CN111311759B CN 111311759 B CN111311759 B CN 111311759B CN 201910071215 A CN201910071215 A CN 201910071215A CN 111311759 B CN111311759 B CN 111311759B
- Authority
- CN
- China
- Prior art keywords
- real image
- video
- augmented reality
- user
- video editing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 92
- 238000012217 deletion Methods 0.000 claims abstract description 25
- 230000037430 deletion Effects 0.000 claims abstract description 25
- 238000000605 extraction Methods 0.000 claims abstract description 6
- 238000010801 machine learning Methods 0.000 claims description 14
- 238000000034 method Methods 0.000 claims description 8
- 230000015572 biosynthetic process Effects 0.000 claims description 7
- 238000003786 synthesis reaction Methods 0.000 claims description 7
- 230000002194 synthesizing effect Effects 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 5
- 238000012937 correction Methods 0.000 claims description 5
- 241000282320 Panthera leo Species 0.000 description 9
- 238000012545 processing Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000003708 edge detection Methods 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000007480 spreading Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/79—Processing of colour television signals in connection with recording
- H04N9/87—Regeneration of colour television signals
- H04N9/8715—Regeneration of colour television signals involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/7837—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04845—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation, expansion or change of colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
- G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B27/036—Insert-editing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/20—Indexing scheme for editing of 3D models
- G06T2219/2004—Aligning objects, relative positioning of parts
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Software Systems (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Human Computer Interaction (AREA)
- Signal Processing (AREA)
- Library & Information Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Architecture (AREA)
- Databases & Information Systems (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Processing Or Creating Images (AREA)
Abstract
There is provided a mobile augmented reality video editing system comprising: a real image input section configured to receive a real image through the image sensor; a space identifier configured to identify a space and a ground within the real image, respectively; an object extraction and display section configured to extract and display an object in a real image; an editing tool display section configured to display a video editing tool to a user; a video processor configured to delete an object touched by a user for deletion among objects extracted and displayed in a real image, and correct a horizontal line of a region constituting the deletion object by replacing each of the horizontal lines of the region constituting the deletion object with a pixel value of the same height in an adjacent space; and a screen display section configured to display the real image corrected by the video processor.
Description
Technical Field
The present invention relates to a system for implementing augmented reality, and more particularly, to a mobile augmented reality video editing system.
Background
With the proliferation of the mobile market and the development of technologies such as computer vision, sensors and video processing, augmented reality technology and virtual reality technology are being improved, and the distribution of applications employing these technologies is accelerating. In addition, as the data processing rate of computers increases, products and various software based on machine learning are gradually spreading, and therefore, products and service items based on image recognition are being mass-produced.
However, existing augmented reality techniques are implemented by taking a picture of the actual environment, overlaying a virtual space on the picture, and then compositing the virtual object into the corresponding space. In other words, since the replacement image is superimposed on the photographed image of the real space, advanced skills are required to edit a specific object or shape photographed, and it takes a long time to edit. Furthermore, in some cases, chroma-key processing or the like requires additional equipment. In addition, when editing an augmented reality video, a worker suffers from the inconvenience of selecting an object or shape to be edited in the video and repeating operations such as resizing and boundary setting.
Therefore, there is a need for a new mobile video editing tool or technique that aims to eliminate the inefficiency of editing an augmented reality video so that even a non-video editing expert can conveniently and quickly acquire his or her desired video.
(related art literature)
(patent literature)
(patent document 1) korean unexamined patent publication No.10-2013-0107404.
(patent document 2) korean patent No.10-1788046.
Disclosure of Invention
1. Technical problem
It is a technical object of the present invention to provide a mobile augmented reality video editing system that enables even a non-video editing expert to edit his or her desired video on a mobile device conveniently and quickly.
Another technical object of the present invention is to provide a mobile augmented reality video editing system capable of obtaining a natural object-deleted video by synthesizing an optimal partial video into a position of a deleted object when deleting an object in an original video to edit an augmented reality video.
It is another technical object of the present invention to provide a mobile augmented reality video editing system that freely changes the position of an object in a video so that the object can be scaled and coordinated with surrounding objects according to the changed position.
It is another technical object of the present invention to provide a mobile augmented reality video editing system capable of implementing augmented reality by replacing an object in a video with a three-dimensional (3D) object that truly and objectively best depicts the object.
It is another technical object of the present invention to provide a mobile augmented reality video editing system that implements augmented reality by synthesizing 3D objects and scaling the 3D objects automatically so that the 3D objects can coordinate with the synthesized video.
2. Technical proposal
A mobile augmented reality video editing system according to an exemplary embodiment of the present invention may be implemented in the form of an application program (app) that may be downloaded and run in a mobile device, and the mobile augmented reality video editing system includes: a real image (real image) input section configured to receive a real image through an image sensor; a space identifier configured to identify a space and a ground within the real image, respectively; an object extraction and display section configured to extract and display an object in the real image; an editing tool display section configured to display a video editing tool to a user; a video processor configured to delete an object touched by a user for deletion among objects extracted and displayed in the real image, and correct a horizontal line of a region constituting the deletion object by replacing each of the horizontal lines of the region constituting the deletion object with a pixel value of the same height as in an adjacent space; and a screen display section configured to display the real image corrected by the video processor.
In the mobile augmented reality video editing system, the video processor may delete an object touched by a user for movement among objects extracted and displayed from a real image, correct the horizontal lines constituting the area of the deleted object by replacing each of the horizontal lines constituting the area of the deleted object with pixel values of the same height as in an adjacent space, adjust a ratio of the object touched for movement using three-dimensional (3D) coordinate values of pixels corresponding to a position designated for movement of the object, thereby generating a corrected video in which the object has been moved.
In addition to the above configuration, another modifiable configuration mobile augmented reality video editing system may further include an object store configured to match an object touched by a user for storing among the extracted and displayed objects with an input file name and store the object in the store.
In this case, the video processor may adjust the scale of the object selected by the user from the storage with the 3D coordinate values of the pixels corresponding to the positions specified for the object synthesis, synthesize the selected object into the real image, and thereby generate the corrected video into which the object has been synthesized.
As another modifiable embodiment, the mobile augmented reality video editing system may further include: an object transmitter configured to transmit an object touched by a user for augmented reality among the extracted and displayed objects to an interoperable external device via a communication network; a service server (service server) configured to compare feature points of an object touched for augmented reality with a machine learning object Database (DB), read a representative 3D object having the highest coincidence rate with the object from the machine learning object DB, and transmit the representative 3D object to the video processor.
In a mobile augmented reality video editing system, a video processor generates a correction video by compositing a representative 3D object into a real image, the scale of which has been adjusted relative to the screen according to the scale value of the object touched for augmented reality.
In addition, the video processor may generate a corrected video in which the size, position, and color of the 3D object have been adjusted according to a user command from the video editing tool.
3. Advantageous effects
According to the above-described technical scheme, the present invention makes it possible to conveniently and rapidly edit a real image acquired from a mobile device by simple manipulation.
Further, the present invention can minimize a difference feeling or an unnatural feeling that can be perceived from an edge portion when editing an object in a real image acquired from a mobile device.
In addition, the present invention automatically recognizes a feature of an object selected by a user among objects in a real image acquired from a mobile device and replaces the object with a standardized, authorized or realistic representative 3D object, so that an augmented reality video can be obtained in a new form.
Furthermore, when the system according to an exemplary embodiment of the present invention is used, a video in which a standardized representative 3D object is inserted into a real image may be obtained. Thus, it is possible to experience a sense of presence, 3D effect, and sense of realism that are perceived when only objects actually present in the real space are automatically recognized and remodeled. In addition, the most realistic, attractive, or wonderful edited video may be obtained according to the characteristics of the representative 3D object.
Drawings
Fig. 1 shows an example of a configuration of a mobile augmented reality video editing system according to an exemplary embodiment of the present invention;
FIG. 2 illustrates an example of a block diagram of the user side mobile augmented reality video editing system of FIG. 1 installed on a mobile device;
FIG. 3 is a flowchart of editing a mobile augmented reality video according to an exemplary embodiment of the present invention;
FIG. 4 illustrates an example of a flow chart for identifying ground and space, respectively, according to an exemplary embodiment of the present invention;
fig. 5A and 5B illustrate examples of real images according to exemplary embodiments of the present invention;
FIGS. 6A and 6B illustrate example views for a comparison between a scene before object deletion and a scene after object deletion;
FIGS. 7A and 7B illustrate example views for a comparison between a scene before object movement and a scene after object movement; and
fig. 8A and 8B show example views for comparison between a scene before the object synthesis process and a scene after the object synthesis process.
Detailed Description
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. A detailed description of known functions or elements related to the present invention will be omitted when it may unnecessarily obscure the gist of the present invention.
Fig. 1 shows an example of a configuration of a mobile augmented reality video editing system according to an exemplary embodiment of the present invention. The mobile augmented reality video editing system according to an exemplary embodiment of the present invention may be classified into a user-side mobile augmented reality video editing system 100 and a service server-side mobile augmented reality video editing system 200, and the user-side mobile augmented reality video editing system 100 may be downloaded and run in a mobile device of a user, while the service server-side mobile augmented reality video editing system 200 provides additional services in combination with the user-side mobile augmented reality video editing system 100.
The user-side mobile augmented reality video editing system 100 deletes objects, moves objects, synthesizes objects, and synthesizes three-dimensional (3D) objects in a real image acquired through at least one image sensor such as a camera provided in a mobile device and displays the processing result on a screen thereof at the request of the mobile device user.
Meanwhile, the service server-side mobile augmented reality video editing system 200 provides services requested by a device user (i.e., service user) in conjunction with the user-side mobile augmented reality video editing system 100. As such a service, the service server-side mobile augmented reality video editing system 200 detects feature points of an object transmitted from the user-side mobile augmented reality video editing system 100, compares the feature points with a machine learning object Database (DB), identifies the object by a 3D object having the highest coincidence rate, reads a representative 3D object representing the 3D object having the highest coincidence rate from the machine learning object DB, and transmits the representative 3D object to the user-side mobile augmented reality video editing system 100. The transmitted 3D object is synthesized into a real image acquired from the user's mobile device and displayed as an augmented reality video.
The configuration of the user-side mobile augmented reality video editing system 100 will be described in more detail below with reference to fig. 2. The user-side mobile augmented reality video editing system 100 described below may be implemented in the form of an application program (app) that may be downloaded to the memory of the mobile device and run. The components of the app interoperate with each other, each component being a set of program code data for performing a specific function, thereby making editing mobile augmented reality video possible.
Fig. 2 shows an example of a block diagram of the user side mobile augmented reality video editing system 100 of fig. 1 installed in a mobile device. As shown, the user-side mobile augmented reality video editing system 100 includes: a real image input section 105 for receiving a real image by the image sensor 100-1; a space identifier 110 for identifying a space within the real image and a ground, respectively; an object extraction and display section 115 for extracting and displaying an object in a real image; an editing tool display section 125 for displaying the video editing tool to the user; a video processor 120 for deleting an object touched by a user for deletion among objects extracted and displayed in a real image, and correcting a horizontal line of a region constituting the deletion object by replacing each of the horizontal lines of the region constituting the deletion object with a pixel value of the same height in an adjacent space; and a screen display part 130 for displaying the real image corrected by the video processor 120.
In these terms, "object touched for deletion" is interpreted as an object touched by the user after the user inputs a "delete" command through the video editing tool, and "object touched for movement" is interpreted as an object touched by the user after the user inputs an "object move" command through the video editing tool. Further, "to store the touched object" is interpreted as an object touched by the user after the user inputs an "object save" command through the video editing tool. When an object is touched after the input of a command by the video editing tool as described above (this corresponds to touch strictly), the touch object is post-processed according to the input command.
The video processor 120 among the elements shown in fig. 2 deletes the object touched by the user for movement among the objects extracted and displayed from the real image, corrects the horizontal line of the area constituting the deletion object by replacing each of the horizontal lines of the area constituting the deletion object with the pixel value of the same height in the adjacent space, and adjusts the scale of the object touched for movement using the 3D coordinate value of the pixel corresponding to the position designated for the object movement, thereby generating a corrected video in which the object has been moved.
In addition to the above system configuration, the user-side mobile augmented reality video editing system 100 according to another exemplary embodiment of the present invention may further include an object store 150 for matching an object touched by a user for storing among the extracted and displayed objects with an input file name and storing the object in the store 100-3. In this case, the video processor 120 adjusts the scale of the object selected by the user from the storage 100-3 with the 3D coordinate values of the pixels corresponding to the position designated for the object synthesis, and synthesizes the selected object into the real image, thereby generating the corrected video into which the object has been synthesized.
In addition to the elements described above, the mobile augmented reality video editing system 100 according to another exemplary embodiment of the present invention further includes: an object transmitter 160 for transmitting an object touched by a user for augmented reality among the extracted and displayed objects to an interoperable external device (i.e., the service server-side mobile augmented reality video editing system 200) via a communication network; and a server for comparing the object touched for augmented reality with the machine learning object DB, reading the 3D object having the highest coincidence rate from the machine learning object DB, and transmitting the 3D object to the video processor 120.
This means that the mobile augmented reality video editing system according to an exemplary embodiment of the present invention can be divided into a user-side mobile augmented reality video editing system 100 and a service server-side mobile augmented reality video editing system 200.
In addition, the video processor 120 generates a corrected video by synthesizing the 3D object into the real image, the proportion of which has been adjusted according to the proportion value of the object touched for augmented reality with respect to the screen.
In addition, the video processor 120 may generate a corrected video in which the size, position, and color of the 3D object have been adjusted according to a user command from the video editing tool.
Elements not depicted in fig. 2 are now described. The image sensor 100-1 corresponding to the video input part generates a real image by scanning and capturing video of a real space. The image sensor 100-1 includes 3D coordinates corresponding to each pixel constituting an image. According to an exemplary embodiment of the present invention, the image sensor 100-1 is shown as a 3D sensor, but one or more cameras capable of acquiring 3D coordinates of each pixel in an image may be used as a video input part.
The User Interface (UI) 100-2 may be implemented as a touch screen having a function of a manipulator for inputting user commands and a function of a display portion for displaying various images and operation states.
In addition to data for driving a plurality of applications of the mobile device, the storage 100-3 stores an object selected by a user, a 3D object, a real image, and the like. The communicator 100-4 includes a technical element for connecting to a mobile communication network and performing short-range communication, and the communicator 100-4 connects to a service server or exchanges data with a nearby device.
In an exemplary embodiment of the present invention, the sensor part 100-5 includes an acceleration sensor, a gyro sensor, etc., and may be used to identify the ground and the space, respectively.
The operation of the mobile augmented reality video editing system 100 including the above elements will be described in more detail below with reference to fig. 3 through 8.
Fig. 3 is a flowchart of editing a mobile augmented reality video according to an exemplary embodiment of the present invention, fig. 4 shows an example of a flowchart of identifying a ground and a space, respectively, according to an exemplary embodiment of the present invention, fig. 5A and 5B show examples of real images according to an exemplary embodiment of the present invention, fig. 6A and 6B show example views for comparison between a scene before object deletion and a scene after object deletion, fig. 7A and 7B show example views for comparison between a scene before object movement and a scene after object movement, and fig. 8A and 8B show example views for comparison between a scene before object synthesis processing and a scene after object synthesis processing.
Referring to fig. 3, first, a person who wants to edit mobile augmented reality video using his or her mobile device downloads and runs a user-side mobile augmented reality video editing system 100 according to an exemplary embodiment of the present invention.
When the mobile augmented reality video editing system 100 is operated, the image sensor 100-1 captures a video of a real space and outputs a real image, and the real image is input to the real image input part 105 of the user side mobile augmented reality video editing system 100 (operation S10). An example of a real image produced by the image sensor 100-1 is shown in fig. 5A.
The space identifier 110, which receives the real image through the real image input section 105, runs a program for identifying the ground and the space, respectively, and identifies the space and the ground, respectively, in the real image. This is now described in further detail with reference to fig. 4.
First, the spatial identifier 110 detects edges in a real image (operation S70). Since a method of detecting an edge of an object in a specific image is a well-known technique in the field of video signal processing, a detailed description thereof will be omitted. When the edge detection technique is applied to a real image, it is possible to extract an object from the real image and detect the boundary of a road, a thing, or the like.
When edge detection is performed on the real image, the spatial identifier 110 generates relative coordinates between feature points constituting the edge based on the position of the mobile device (corresponding to the reference point) (operation S75). Subsequently, a plane is generated using two of the reference point and the generated relative coordinates (operation S80), and a plane parallel to the ground among the generated planes is identified as the ground (operation S85). In this way, the ground and space are identified separately.
Meanwhile, the object extraction and display part 115 extracts an object having a shape that can be detected by an edge from the real image, the ground and the space of which are respectively identified, and displays the extracted object in the UI 100-2 (operation S20). As shown in fig. 5B, in order to increase the visibility of the extracted object, the edge of the extracted object may be bold or the visibility of the object area may be opened.
When the object is extracted and displayed in the real image, the user invokes a video editing tool for video editing, and the editing tool display section 125 displays the video editing tool in response to the invocation (operation S25). The video editing tool includes a "delete" command for deleting an object, an "object move" command for moving an object, and a "save" command for storing an object. These are merely examples and may further include various tools for video editing.
When the video editing tool is displayed, the user inputs (or touches) a command that he or she wants. For example, when the user wants to delete an object, the user inputs a "delete" command and then touches the object to be deleted. When a deletion command is input through the video editing tool and a corresponding object is touched (operation S30), the video processor 120 deletes an object touched by the user for deletion among objects extracted and displayed in the real image as shown in fig. 5B (operation S35), as shown in fig. 6A, and corrects the horizontal line of the area constituting the deletion object by replacing each of the horizontal lines of the area constituting the deletion object with pixel values of the same height (off the ground) in the adjacent space (operation S40).
When the video processor 120 completes the processing of the object for which the user has requested editing, the screen display part 130 displays a real image obtained by turning off the visibility of the object area as shown in fig. 6B as a real image corrected by the video processor 120 (operation S45).
Referring to fig. 6B and 5A in comparison, a user can conveniently edit the real image shown in fig. 5A (as shown in fig. 6B) simply by invoking a video editing tool in his or her mobile device, inputting a delete command with a touch, and selecting an object to delete.
Further, the horizontal lines in the area where the object has been deleted are corrected by replacing each of the horizontal lines in the area where the object has been deleted with pixel values of the same height (from the ground) in the space adjacent to the object area. Accordingly, the pixel value of the region of the deletion object coordinates with the pixel value of the adjacent space, and a natural video can be obtained. In addition, since the edge lines of the ground and the object are replaced with the pixel values of the same horizontal line, it is possible to minimize the sense of difference or the sense of editing that can be perceived from the edge portion.
As an example, the exemplary embodiment of the present invention shows that each horizontal line is simply replaced with pixel values of the same height in adjacent spaces. However, when pixel values of the same height in adjacent spaces have patterns, the corrected areas need to be corrected to reflect the patterns.
The operation of processing the object in the real image according to the video editing command different from the object deletion will be described in detail below.
When an object movement command, an object touch, and a position designated for object movement are sequentially input for a real image by a video editing tool while the system 100 is running (operation S60), the video processor 120 deletes the object touched for movement by the user among the objects extracted and displayed in operation S20, corrects the horizontal line of the area constituting the deleted object by replacing each of the horizontal lines of the area constituting the deleted object with pixel values of the same height in the adjacent space, adjusts the ratio of the object touched for movement using 3D coordinate values of the pixels corresponding to the position designated for object movement (operation S60), more specifically, the depth coordinate (z axis, assuming that the x axis is the horizontal axis and the y axis is the vertical axis), which gives the stereoscopic.
A corrected video obtained by moving an object present in a real image (fig. 5A) is shown in fig. 7A. Compared to fig. 5A, the corrected video of fig. 7A shows that the tree as the second object on the right side of the road has moved rightward. Since the proportion of the object touched for movement has been adjusted according to the depth coordinates of the pixels at the position specified for the object movement even in correcting the video, a scene in which the video in which the object touched for movement is coordinated with surrounding objects or the background has been edited can be obtained. For reference, fig. 7B shows a finally displayed scene obtained by the screen display portion 130 turning off the visibility of the object area.
Meanwhile, when an object copy command, an object touch, and a position designated for object copy are sequentially input through the video editing tool, the video processor 120 generates a correction video in which an object touched by the user for copy is scaled using depth coordinates and pasted to the position designated for object copy among the objects extracted and displayed in operation S20, as shown in fig. 8A. Fig. 8A is a scene obtained by copying a tree as a first object on the right side of a road in a real image (fig. 5A) to the left.
As described above, the user-side mobile augmented reality video editing system 100 according to an exemplary embodiment of the present invention enables a user to conveniently and rapidly edit a real image acquired through the image sensor 100-1 with a simple touch operation on a mobile device.
In addition, according to the present invention, the video is corrected using the pixel values of the edge line of the object region or the vicinity region, and thus a natural edited video can be obtained.
In addition to the video editing described above, a user may also compose additional stored objects into one real image using the mobile augmented reality video editing system 100 according to an exemplary embodiment of the present invention.
For example, the object store 150 matches an object touched by the user for storing (an object interpreted as being touched after inputting the save command) among the objects extracted and displayed in operation S20 with a file name input by the user and stores the object in the store 100-3. Then, according to the user's composition command, the video processor 120 may generate a corrected image by adjusting the scale of the object selected by the user from the storage 100-3 with the 3D coordinate values of the pixels corresponding to the position specified for the object composition and compositing the object into a real image. As described above, such corrected video is displayed by the screen display section 130.
Further, the mobile augmented reality video editing system 100 according to an exemplary embodiment of the present invention may synthesize a virtual object into a real image acquired through the image sensor 100-1 and display an augmented reality image, and may synthesize an optimal 3D object obtained through machine learning into a real image and display an augmented reality image. Here, the optimal 3D object obtained through machine learning may be a 3D object identified as a real objective, standardized 3D object or an authorized 3D object created through machine learning.
Due to limitations in location, time, cost, etc., it may be difficult for an average person to edit the optimal augmented reality video. When an augmented reality video can be edited by replacing an object obtained from a real image with an object that has been identified as real, objective, standardized, or the like, the truest, beautiful, or wonderful edited video can be created without infringing copyrights.
Accordingly, the object transmitter 160 of the mobile augmented reality video editing system 100 according to an exemplary embodiment of the present invention transmits an object touched for augmented reality by the user (an object interpreted as being touched after inputting an augmented reality command) among the objects extracted and displayed in operation S20 to the interoperable service server-side mobile augmented reality video editing system 200 via a communication network.
For example, assuming that the object touched for augmented reality is a lion dog in a real image and a scene where a host walks with the lion dog tethered by a belt is included in the real image, the shape of the lion dog or feature points of the shape of the lion dog are transmitted to the service server-side mobile augmented reality video editing system 200.
Then, the service server-side mobile augmented reality video editing system 200 detects feature points that are the shape of an object touched by augmented reality (i.e., a lion dog), compares the detected feature points with the machine learning object DB, and identifies a 3D object having the highest coincidence rate. As a result, the service server may identify the received object as a lion dog and send a representative 3D object of the identified object to the video processor 120 of the user-side mobile augmented reality video editing system 100.
Assuming that the representative 3D object received from the service server is a 3D shape of a lion dog that is a dominant year of the lion dog, the video processor 120 corrects the real image by synthesizing the received representative 3D object into the real image. In this case, the video processor 120 may generate the corrected video by adjusting the scale of the representative 3D object with respect to the screen according to the scale value of the object touched for augmented reality in the previous real image, so that the natural edited video may be obtained. The video processor 120 may also generate corrected video that has been resized, positioned, and color of the representative 3D object in accordance with user commands from the video editing tool.
When the user-side mobile augmented reality video editing system 100 of the present invention is configured to interoperate with the machine learning object DB located at the service server side as described above, the service user can obtain an edited augmented reality video in which the owner accompanies a dog (which is a annual lion dog prize master).
This is merely an example, and a system according to an exemplary embodiment of the present invention may be used to obtain video in which a standardized representative 3D object is inserted into a real image. Thus, the system can be used as a tool to experience the sense of presence, 3D effects and realism that are perceived when only objects actually present in real space are automatically identified and remodeled.
Although the invention has been described above with reference to the embodiments shown in the drawings, it will be understood by those of ordinary skill in the art that these embodiments are exemplary and that various modifications and equivalents may be made thereto. Accordingly, the technical scope of the present invention should be determined by the following claims.
Claims (6)
1. A mobile augmented reality video editing system comprising:
a real image input section configured to receive a real image through the image sensor;
a space identifier configured to identify a space and a ground within the real image, respectively;
an object extraction and display section configured to extract and display an object in the real image;
an editing tool display section configured to display a video editing tool to a user;
an object transmitter configured to transmit an object touched by the user for augmented reality among the extracted and displayed objects to an interoperable external device via a communication network;
a service server configured to: comparing feature points of an object touched for augmented reality with a machine learning object database, reading a representative 3D object having the highest coincidence rate with the object from the machine learning object database, and transmitting the representative 3D object to a video processor;
the video processor configured to correct the real image by synthesizing the transmitted representative 3D object into the real image; and
a screen display section configured to display the real image corrected by the video processor,
wherein the video processor is further configured to: the method includes deleting an object touched by the user for deletion among objects extracted and displayed in the real image, and correcting the horizontal lines of the area constituting the deletion object by replacing each of the horizontal lines of the area constituting the deletion object with a pixel value of the same height as that of the horizontal lines in the adjacent space.
2. The mobile augmented reality video editing system of claim 1,
wherein the video processor deletes an object touched by the user for movement among the extracted and displayed objects from the real image, corrects the horizontal lines constituting the area of the deleted object by replacing each of the horizontal lines constituting the area of the deleted object with a pixel value of the same height as the pixel value of the horizontal line in the adjacent space, adjusts the scale for moving the touched object using the three-dimensional 3D coordinate values of the pixels corresponding to the position designated for object movement, thereby generating a corrected video in which the object has been moved.
3. The mobile augmented reality video editing system of claim 1,
also included is an object store configured to match an object touched by the user for storing among the extracted and displayed objects with an input file name and store the object in the store,
wherein the video processor adjusts the scale of the object selected by the user from the storage with 3D coordinate values of pixels corresponding to the positions specified for object synthesis, synthesizes the selected object into the real image, thereby producing a corrected video into which the object has been synthesized.
4. The mobile augmented reality video editing system of claim 1,
wherein the video processor generates a correction video by synthesizing the representative 3D object into the real image, the proportion of the correction video having been adjusted with respect to the screen according to a proportion value of the object touched for augmented reality.
5. The mobile augmented reality video editing system of claim 1 or 4,
wherein the video processor generates a correction video in which the size, position and color of the representative 3D object have been adjusted according to a user command from the video editing tool.
6. The mobile augmented reality video editing system of any one of claim 1 to 4,
wherein each of the real image input section, the spatial identifier, the object extraction and display section, the editing tool display section, the video processor, the screen display section, and the object transmitter is a set of program code data constituting an application app that can be downloaded to a memory of a user mobile device and run.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR10-2018-0160189 | 2018-12-12 | ||
| KR1020180160189A KR102091217B1 (en) | 2018-12-12 | 2018-12-12 | Augmented reality video editing system for a mobile device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111311759A CN111311759A (en) | 2020-06-19 |
| CN111311759B true CN111311759B (en) | 2024-02-20 |
Family
ID=69957244
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910071215.5A Active CN111311759B (en) | 2018-12-12 | 2019-01-25 | Mobile augmented reality video editing system |
Country Status (3)
| Country | Link |
|---|---|
| US (1) | US10643662B1 (en) |
| KR (1) | KR102091217B1 (en) |
| CN (1) | CN111311759B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11366983B2 (en) * | 2020-09-09 | 2022-06-21 | International Business Machines Corporation | Study-level multi-view processing system |
| KR102678078B1 (en) * | 2021-02-08 | 2024-06-26 | 한국가스공사 | Apparatus and method for providing underground facilities management service |
| KR102431890B1 (en) | 2021-12-02 | 2022-08-12 | 주식회사 미리디 | Method of providing a three dimensional sample view based on modeling data of differential quality rendered on a server, and apparatus therefor |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101520904A (en) * | 2009-03-24 | 2009-09-02 | 上海水晶石信息技术有限公司 | Reality augmenting method with real environment estimation and reality augmenting system |
| KR20100014198A (en) * | 2008-07-31 | 2010-02-10 | (주)지아트 | Method and system for authoring 3d model object in augmented reality environment |
| CN102867169A (en) * | 2011-04-08 | 2013-01-09 | 索尼公司 | Image processing apparatus, display control method and program |
| CN103390287A (en) * | 2012-05-11 | 2013-11-13 | 索尼电脑娱乐欧洲有限公司 | Apparatus and method for augmented reality |
| CN103685728A (en) * | 2012-09-21 | 2014-03-26 | Lg电子株式会社 | Mobile terminal and controlling method thereof |
| KR20150091798A (en) * | 2014-02-04 | 2015-08-12 | 엘지전자 주식회사 | Terminal and method for combining images thereof |
| JP2016099638A (en) * | 2014-11-18 | 2016-05-30 | セイコーエプソン株式会社 | Image processing apparatus, image processing apparatus control method, and computer program |
| KR20180000017A (en) * | 2016-06-21 | 2018-01-02 | 한양대학교 에리카산학협력단 | Augmented reality providing mehtod using smart glass |
| WO2018039270A1 (en) * | 2016-08-22 | 2018-03-01 | Magic Leap, Inc. | Virtual, augmented, and mixed reality systems and methods |
| KR101923507B1 (en) * | 2018-01-11 | 2018-11-29 | 김승은 | Method for providing interactive story-telling contents based on augmented reality |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8549554B2 (en) * | 2006-03-07 | 2013-10-01 | Sony Computer Entertainment America Llc | Dynamic replacement of cinematic stage props in program content |
| US8520935B2 (en) * | 2010-02-04 | 2013-08-27 | Sony Corporation | 2D to 3D image conversion based on image content |
| KR101788046B1 (en) | 2010-11-03 | 2017-10-19 | 엘지전자 주식회사 | Mobile terminal and method for controlling the same |
| US10019962B2 (en) * | 2011-08-17 | 2018-07-10 | Microsoft Technology Licensing, Llc | Context adaptive user interface for augmented reality display |
| US9626798B2 (en) * | 2011-12-05 | 2017-04-18 | At&T Intellectual Property I, L.P. | System and method to digitally replace objects in images or video |
| KR20130107404A (en) | 2012-03-22 | 2013-10-02 | 한남대학교 산학협력단 | An augmented object fusion system using media contents edit and a method thereof |
| US9286727B2 (en) * | 2013-03-25 | 2016-03-15 | Qualcomm Incorporated | System and method for presenting true product dimensions within an augmented real-world setting |
| KR101876419B1 (en) * | 2016-01-07 | 2018-07-10 | 한국전자통신연구원 | Apparatus for providing augmented reality based on projection mapping and method thereof |
| TWI576787B (en) * | 2016-02-05 | 2017-04-01 | 黃宇軒 | Systems and applications for generating augmented reality images |
| US10373380B2 (en) * | 2016-02-18 | 2019-08-06 | Intel Corporation | 3-dimensional scene analysis for augmented reality operations |
| US10816334B2 (en) * | 2017-12-04 | 2020-10-27 | Microsoft Technology Licensing, Llc | Augmented reality measurement and schematic system including tool having relatively movable fiducial markers |
| US10008045B1 (en) * | 2017-12-21 | 2018-06-26 | Capital One Services, Llc | Placement of augmented reality objects using a bounding shape |
-
2018
- 2018-12-12 KR KR1020180160189A patent/KR102091217B1/en active IP Right Grant
-
2019
- 2019-01-16 US US16/249,227 patent/US10643662B1/en active Active
- 2019-01-25 CN CN201910071215.5A patent/CN111311759B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20100014198A (en) * | 2008-07-31 | 2010-02-10 | (주)지아트 | Method and system for authoring 3d model object in augmented reality environment |
| CN101520904A (en) * | 2009-03-24 | 2009-09-02 | 上海水晶石信息技术有限公司 | Reality augmenting method with real environment estimation and reality augmenting system |
| CN102867169A (en) * | 2011-04-08 | 2013-01-09 | 索尼公司 | Image processing apparatus, display control method and program |
| CN103390287A (en) * | 2012-05-11 | 2013-11-13 | 索尼电脑娱乐欧洲有限公司 | Apparatus and method for augmented reality |
| CN103685728A (en) * | 2012-09-21 | 2014-03-26 | Lg电子株式会社 | Mobile terminal and controlling method thereof |
| KR20150091798A (en) * | 2014-02-04 | 2015-08-12 | 엘지전자 주식회사 | Terminal and method for combining images thereof |
| JP2016099638A (en) * | 2014-11-18 | 2016-05-30 | セイコーエプソン株式会社 | Image processing apparatus, image processing apparatus control method, and computer program |
| KR20180000017A (en) * | 2016-06-21 | 2018-01-02 | 한양대학교 에리카산학협력단 | Augmented reality providing mehtod using smart glass |
| WO2018039270A1 (en) * | 2016-08-22 | 2018-03-01 | Magic Leap, Inc. | Virtual, augmented, and mixed reality systems and methods |
| KR101923507B1 (en) * | 2018-01-11 | 2018-11-29 | 김승은 | Method for providing interactive story-telling contents based on augmented reality |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111311759A (en) | 2020-06-19 |
| US10643662B1 (en) | 2020-05-05 |
| KR102091217B1 (en) | 2020-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2021203688B2 (en) | Volumetric depth video recording and playback | |
| CN107251101B (en) | Scene modification for augmented reality using markers with parameters | |
| US8644467B2 (en) | Video conferencing system, method, and computer program storage device | |
| US7965304B2 (en) | Image processing method and image processing apparatus | |
| JP2020535509A (en) | Methods, devices and systems for automatically annotating target objects in images | |
| JP5093053B2 (en) | Electronic camera | |
| JP3148045B2 (en) | 3D object CG creation device | |
| US20080018668A1 (en) | Image Processing Device and Image Processing Method | |
| US20140181630A1 (en) | Method and apparatus for adding annotations to an image | |
| KR20190045317A (en) | Image processing apparatus, image generation method, and computer program | |
| CN111311759B (en) | Mobile augmented reality video editing system | |
| US20190130648A1 (en) | Systems and methods for enabling display of virtual information during mixed reality experiences | |
| KR20140082610A (en) | Method and apaaratus for augmented exhibition contents in portable terminal | |
| JP7208549B2 (en) | VIRTUAL SPACE CONTROL DEVICE, CONTROL METHOD THEREOF, AND PROGRAM | |
| KR101875047B1 (en) | System and method for 3d modelling using photogrammetry | |
| WO2014094874A1 (en) | Method and apparatus for adding annotations to a plenoptic light field | |
| KR20210032549A (en) | Image processing apparatus, image processing method, and computer program | |
| US20190272658A1 (en) | Image processing device, image processing method, and program | |
| KR102201285B1 (en) | System for editing an augmented reality image | |
| US11030359B2 (en) | Method and system for providing mixed reality service | |
| JP6011567B2 (en) | Information processing apparatus, control method thereof, and program | |
| WO2019012314A1 (en) | Method of displaying a wide-format augmented reality object | |
| CN111640190A (en) | AR effect presentation method and apparatus, electronic device and storage medium | |
| JP2017084215A (en) | Information processing system, control method thereof, and program | |
| JP2020074108A (en) | Information processing system, control method thereof, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |