KR20210079029A - Method of recording digital contents and generating 3D images and apparatus using the same - Google Patents

Abstract

The present invention relates to a method of recording digital content and generating a 3D image. The method includes the steps of: transmitting a captured image including the same target object or object of interest from at least one or more plurality of connected devices to one of the plurality of connected devices; receiving camera shooting condition information including at least one of the corrected position of the current device, the position of the target object, the shooting direction, the image size, and the focal length from one of the plurality of connected devices; generating digital content in consideration of the received camera shooting condition information; and transmitting the generated digital content to one of the plurality of connected devices. The corrected position of the current device is determined in consideration of the position of the target object or the object of interest in the captured image. The shooting direction is parallel to a line connecting the position of the target object and the corrected position of the current device. The focal length is determined according to the distance between the position of the target object and the position of the corrected current. The at least one or more of the plurality of connected devices generates the digital content with the image size included in the shooting condition information. By acquiring image data in various shooting directions using multiple connected devices, it is possible to effectively create 360-degree 3D data.

Description

Method of recording digital contents and generating 3D images and apparatus using the same}

The present invention relates to a method of generating a 3D image by recording digital content, and more particularly, digital content capable of effectively producing 360-degree 3D data by acquiring image data in various shooting directions using several connected devices. To a method and apparatus for recording a 3D image.

Conventionally, a large amount of distortion occurred because panoramic photos were taken in the left and right directions and up and down, arranged in a circular sphere shape, and the screen displayed on the wall of the sphere was viewed while standing at the camera position in the middle of the sphere.

In addition, the user is fixed at the camera position, can only look left, right, up, and down by turning his head, and it is difficult to move the camera to a position different from the camera shooting position in the image.

Accordingly, there is an increasing need to reduce distortion and provide an image as viewed from multiple photographing points by simultaneously photographing the camera at several locations.

On the other hand, in the case of building 3D data by shooting a 360-degree image, it takes a lot of money to collect data. In addition, when using videos shot in various shooting directions, it takes a long time to collect, there is a problem of overlapping regions when stitching tile images, and a high-resolution image of the region of interest is required. Therefore, there is a need for a method for effectively producing 360-degree 3D data by acquiring image data in various shooting directions using various devices.

Prior art 1 (KR 10-2017-0017700A) relates to an electronic device that generates a 360-degree 3D stereoscopic image, generates a left-eye spherical image and a right-eye spherical image, and uses the generated left/right spherical image to obtain depth information. It is characterized in that the three-dimensional effect is adjusted using the acquired and acquired depth information.

Prior art 2 (KR 10-1639275B1) is a 3D spherical image generated by a 360-degree rendering processing unit through a user interface unit so that a user can view a free-view image as desired by displaying a 360-degree image up, down, left, and right. , it is characterized by processing to perform the zoom function, but a specific configuration is not shown.

Therefore, the first problem to be solved by the present invention is a method of generating 3D images by recording digital content that can effectively produce 360-degree 3D data by acquiring image data in various shooting directions using several connected devices. will provide

A second object to be solved by the present invention is to provide an apparatus for generating a 3D image by recording digital content capable of reducing distortion and providing an image as viewed from multiple shooting points by simultaneously shooting with cameras at multiple locations .

Another object of the present invention is to provide a computer-readable recording medium in which a program for executing the above-described method is recorded on a computer.

In order to achieve the first object, the present invention includes: transmitting a captured image including the same target object or object of interest from at least one or more connected devices to one of the plurality of connected devices; receiving camera shooting condition information including at least one of a corrected position of the current device and a position of the target object, a shooting direction, an image size, and a focal length from one of the plurality of connected devices; generating digital content in consideration of the received camera shooting condition information; and transmitting the generated digital content to one of the plurality of connected devices, wherein the corrected position of the current device is determined in consideration of a target object position or an object of interest position in the captured image; , the photographing direction is parallel to a line connecting the position of the target object and the corrected position of the current device, and the focal length is determined according to a distance between the position of the target object and the corrected current position, and the at least one or more A plurality of connected devices provides a method of generating digital content with an image size included in the shooting condition information.

According to an embodiment of the present invention, it is preferable that the captured image includes a marker for detecting the target object, and the resolution of the digital content is determined according to a focal length.

In addition, the digital content preferably includes head or eye tracking information of the user.

According to another embodiment of the present invention, the method comprising: determining a current location using a captured image of an object of interest; Camera shooting condition information including at least one of the current position and the corrected position, shooting direction, image size, and focal length of the one or more connected devices to at least one connected device among a plurality of connected devices transmitting; receiving digital content captured with the same image size by using the transmitted camera shooting condition information from at least one or more connected devices among the plurality of connected devices; and the corrected position is determined based on the captured images, and the photographing direction is parallel to an extension line connecting the current position and the corrected position.

The plurality of connected devices may be classified into groups to receive digital content having different image sizes.

Detect a marker attached to the plurality of connected devices in the captured image, and receive tracking information of the head or eyes from at least one of the connected devices.

According to another embodiment of the present invention, the method comprising: transmitting image capturing event information including information of a moving path and target object for storing digital content to a master device; transmitting object of interest information including 3D coordinates and color information of the object of interest to the master device; calculating the current position of the master device and corrected positions of connected devices, and transmitting the calculation result and camera shooting condition information to the master device; receiving, through the master device, a captured image captured by the connected devices using the camera shooting condition information; generating a 3D image by using the received digital content, wherein a photographing direction of the captured images is parallel to an extension line connecting a position of the target object and a position of at least one of connected devices It provides a method for generating 3D images in a server among connected devices.

The camera shooting condition information may include a focal length, an image size, corrected positions of the connected devices, and a position of the target object, and an image resolution may be determined based on the focal length information.

In addition, it is preferable to generate a 3D image after classifying the received captured images into different groups according to the image size included in the camera shooting condition information.

On the other hand, with respect to the captured image having the same image size included in the camera shooting condition information, a fitting plane is determined between captured captured images so that a change in coordinate information of an object of interest existing in an overlapping area is minimized. can

The method may further include determining a location of a virtual camera connecting the captured images.

In order to achieve the second object, the present invention includes: a transmitter configured to transmit a captured image including the same target object or object of interest from at least one or more connected devices to one of the plurality of connected devices; a receiver configured to receive camera shooting condition information including at least one of a corrected position of the current device, a position of the target object, a shooting direction, a focal length, and an image size from one of the plurality of connected devices; and a photographing unit generating digital content in consideration of the received camera photographing condition information, wherein the transmitting unit transmits the generated digital content to one of the plurality of connected devices, and the currently corrected position is It is determined in consideration of a target object position or an object of interest position in the captured image, the shooting direction is parallel to a line connecting the position of the target object and the currently corrected position, and the focal length is the target object position and the Provided is a connected device that generates digital content determined according to a distance between corrected current locations.

According to an embodiment of the present invention, a calculator for determining a current location using a captured image of an object of interest; Transmitting camera shooting condition information including at least one of the current position, the corrected position of the one or more connected devices, a shooting direction, a focal length, or an image size to at least one or more connected devices among a plurality of connected devices a transmitter to; a receiver configured to receive digital content generated by using the camera shooting condition information from at least one connected device among the plurality of connected devices; and the current location is determined by comparing a boundary line or color between the object of interest information and the captured image, and the photographing direction is parallel to an extension line connecting the current location and the corrected location of the one or more connected devices. , wherein the corrected position of the one or more connected devices is determined based on the captured images.

According to another embodiment of the present invention, there is provided an event information generating unit for generating image capturing event information including a moving path and target object information for storing digital content; Transmits the image capturing event information to the master device, transmits object of interest information including 3D coordinates and color information of the object of interest to the master device, and calculates the current position of the master device and corrected positions of connected devices and a transmitter for transmitting the calculation result and camera shooting condition information; a receiver configured to receive captured images generated using camera shooting condition information in the connected devices through the master device; and a 3D image generator configured to generate a 3D image by grouping the received digital content by digital content by using the image size information included in the photographing information, wherein the photographing direction of the captured images is the position of the target object. And it provides a server for generating a 3D image, characterized in that parallel to the extension line connecting the position of at least one of the connected devices.

In order to solve the other technical problem, the present invention provides a computer-readable recording medium in which a program for executing a method of generating a 3D image by recording the digital content described above in a computer is recorded.

According to the present invention, it is possible to effectively produce 360-degree 3D data by acquiring image data in various shooting directions using various connected devices.

In addition, according to the present invention, by simultaneously photographing by cameras in several locations, distortion can be reduced and an image as viewed from several photographing points can be provided.

1 is an example of capturing images in a vehicle as a master device and a plurality of mobile slave devices according to an embodiment of the present invention.
2 is a block diagram of an apparatus for generating a 3D image by recording digital content according to an embodiment of the present invention.
3 illustrates changes in a focal length and a photographing angle (AOV) when an object is photographed while maintaining the same photographing field of view (FOV).
4 shows an example of a color marker used for easy recognition between connected devices.
5 is a diagram illustrating a state in which a plurality of vehicles are wirelessly connected to a plurality of mobile devices for digital content recording according to another embodiment of the present invention.
6 illustrates an example of stitching a plurality of images in order to process a captured image into a 3D image.
7 is a flowchart of a method of recording digital content in a slave device among connected devices according to an exemplary embodiment of the present invention.
8 is a flowchart of a method of generating a 3D image in a master device among connected devices according to an exemplary embodiment of the present invention.
9 is a flowchart of a method of generating a 3D image in a server among connected devices according to an exemplary embodiment of the present invention.

Prior to the description of the specific content of the present invention, for the convenience of understanding, the outline of the solution to the problem to be solved by the present invention or the core of the technical idea is presented first.

A plurality of connected devices may capture the same object as an image having different focal lengths at the same time. In this case, the captured image can be effectively converted into 3D image data only when the captured image is captured with the same focal length and the same field-of-view (FOV). In order to extract accurate focal coordinates, a panoramic image having overlapping regions may be synthesized by using 3D location information of a point of interest (POI) existing in a captured image.

Also, images of different resolutions for the same object of interest may be captured by connected devices. For example, in the case of photographing with a camera under the same conditions, an image photographed near the object of interest has a higher resolution than an image photographed farther away. And the closer to the object, the smaller the angle-of-view (AOV) is. Therefore, it is necessary to adjust the imaging conditions of the connected device to satisfy a consistent imaging field of view and resolution for the same object of interest.

The coordinates of the connected device use the obtained 3D location information or a photographed image, and a more accurate localization is possible using the 3D location information and color information of the main object of interest information stored on the road map of the precision map and the DB. It is possible.

Therefore, by using the current location information of the master device obtained through the location calculation and real-time image information received from the slave devices around the master device, the location of the surrounding slave device can be calculated, and the result can be transmitted to the surrounding slave device. . In addition, recognition between connected devices can be recognized through a wireless signal or a physical marker existing on the device. Whether to capture an image and whether to transmit/receive camera photographing condition information between a plurality of devices may be determined according to whether the marker is recognized.

The camera shooting condition information includes information provided to a user and a camera control for acquiring image information. In this case, the information provided to the user includes augmented-reality (AR), virtual-reality (VR), and navigation information, and the user of the device effectively controls or reacts to the device in response to the information. Additional information can be requested by sending a signal.

In this case, the camera shooting condition information includes a camera shooting direction, a shooting field of view, an image size, a focal length, and an image resolution of a mobile slave device that allows the peripheral mobile slave device to be shot in the same direction as the shooting direction (viewpoint) of the master device. .

In this case, when the image of the mobile slave device is captured by a person, the master device may receive and analyze the head or eye movement information of the photographer, and may provide display information to the mobile slave device.

In order to join images extracted from multiple devices, location information of the devices is required. The location information calculation of the device is performed on a cloud server or a local hub device or virtual machine connected to the cloud server, or a device with sufficient available resources among local mobile devices is selected as the master device. The position value of the master device may be calculated using images collected from a plurality of devices, 3D coordinates of an object of interest (landmark or POI), edge information, or precision road map information.

In addition, the master device may receive front/rear/left/right images obtained by photographing the master device as a target object from the slave device, and calculate the slave device location using network location information of location information of a plurality of devices. It is preferable that the master device location, the slave device location, and the location of a specific object of interest exist in the same shooting direction in the image.

In addition, the master device and the slave device may include physical marker information in order to effectively recognize the locations of the devices in images taken by a plurality of devices.

Images captured by multiple devices may be transmitted to a server (hub device or virtual machine) through a master device, and 3D image data may be generated in another cloud server. In this case, after generating a spliced tile image for each image group having the same imaging field, 3D image data may be produced by rejoining different tile images based on a virtual camera focus.

Hereinafter, with reference to the accompanying drawings, a preferred embodiment in which a person of ordinary skill in the art to which the present invention pertains can easily practice the present invention will be described in detail. However, these examples are for explaining the present invention in more detail, it will be apparent to those skilled in the art that the scope of the present invention is not limited thereby.

The configuration of the invention for clarifying the solution of the problem to be solved by the present invention will be described in detail with reference to the accompanying drawings based on a preferred embodiment of the present invention, but the same in assigning reference numbers to the components of the drawings For the components, even if they are on different drawings, the same reference numbers are given, and it is noted in advance that the components of other drawings can be cited when necessary in the description of the drawings. In addition, when it is determined that detailed descriptions of well-known functions or configurations related to the present invention and other matters may unnecessarily obscure the gist of the present invention in explaining the operating principle of the preferred embodiment of the present invention in detail, A detailed description thereof will be omitted.

1 is an example of capturing images in a vehicle serving as a master device and a plurality of mobile slave devices according to an embodiment of the present invention.

Referring to FIG. 1 , an environment for generating 3D image information by simultaneously acquiring images having multiple focal points using a plurality of adjacent mobile devices connected to a vehicle and a network and stitching the images is shown.

In one embodiment of the present invention, an image having a plurality of focal points may be acquired from a camera disposed in front/rear/left/right of a vehicle, and an image may be acquired by simultaneously operating the front/rear camera in a mobile device. In addition, if the mobile device is an automated device such as a robot, it is possible to acquire an automated image by mounting a multi-camera.

In the case of a mobile device, the user can take an image according to the movement of a predefined master device, a master vehicle, or use camera shooting condition information sent from the master device to the user's mobile device. After the mobile device and the master device vehicle are connected to the network, the master device vehicle may recognize the mobile device or, conversely, the mobile device may recognize the master device vehicle.

In this case, the recognition between each device may be recognized by mixing a network connection and an image recognition method. For example, a connection may be made only by network connection or image information, or the connection may be sequentially approved.

It is preferable to photograph the surroundings based on the photographing direction in which the master apparatus vehicle is the target object as a criterion for photographing images in a plurality of devices. Therefore, it is preferable that the target object, that is, the master device vehicle and the mobile device secure a view from each other, or that the mobile device reflects the location information of the target object in real time to photograph it.

In addition, each mobile device needs to acquire an image that has the same photographing field of view and satisfies a predetermined resolution. To this end, using distance information between the master device vehicle and the mobile device, it may be performed by adjusting the focal length and the photographing resolution of the mobile device.

Referring to FIG. 1 , a vehicle 100 , which is a master device, and neighboring mobile devices 110 , 120 , 130 , 140 , 150 and 160 are connected through a network to acquire image information.

The image taken in the front direction of the master device vehicle 100 is denoted by A, the image taken in the left direction is denoted by B, the image shot in the right direction is denoted by C, and the image shot in the rear direction is denoted by D.

The A, B, C, and D images are images captured by the master device vehicle, and have a greater angle of view (AOV) than the images captured by the mobile devices 110 to 160 . In the case of image B including an object located on the left side of the master device vehicle 100, the resolution of the image of the mobile device 120 captured at a close distance is higher than the resolution of the image captured by the master device vehicle 100 and the shooting angle is small. can

In this case, a wide shooting angle image can be provided using the image captured by the master device vehicle 100 , and when a specific object of interest is zoomed in, the image is captured by the mobile device as shown on the right side of FIG. 1 . One image may be used to provide the same resolution image or high resolution image.

In this case, it is preferable that images photographed by a plurality of mobile devices have the same screen size (including a photographing field of view and frame size) for image splicing, and satisfy a certain resolution. However, since the distances between the plurality of mobile devices and the master device vehicle 100 are all different, in consideration of the distance differences between these mobile devices and the master device vehicle 100 , the focal-length of each mobile device is It is desirable to provide a 3D image that satisfies a predetermined resolution by adjusting the and image resolution.

The focal length and resolution of the mobile device may be adjusted within a predetermined range using a delta value (distance difference (i) - distance difference (j)) between the distance differences. The distance difference i means the distance between the mobile device i and the master device vehicle 100 .

In addition, when the distance difference is out of a certain range, it is preferable to exclude the image captured by the mobile device when joining the 3D image.

Meanwhile, the method of adjusting the focal length is as follows.

where I is the on-image distance between the image and the mobile device, O is the object distance between the object and the mobile device, and f is the focal length.

In order to photograph an object with the same photographing field of view and frame size in multiple mobile devices, the focal length should be increased as the object distance (O) increases, and a certain resolution can be maintained only when photographing at a high resolution (high resolution photographing is an image sensor size) means large). Therefore, as the mobile device and the master device vehicle 100 are closer, the distance O between the mobile device and the object is relatively large, so the focal length of the camera should be increased and the image resolution should be relatively high.

2 is a block diagram of an apparatus for generating a 3D image by recording digital content according to an embodiment of the present invention.

Referring to FIG. 2 , the device for generating a 3D image according to the present embodiment includes a master device 200 , a server 210 , and a slave device 220 .

The master device 200 includes a calculator 201 , a transmitter 202 , a receiver 203 , and a 3D image generator 204 .

The calculator 201 compares the object of interest information with the captured images of the object of interest to determine the current location of the master device. Preferably, the current location is determined by comparing a boundary line or color between the object of interest information and the captured images. In addition, it is possible to detect a marker attached to the slave device in the captured image.

The transmitter 202 is at least one slave device among the plurality of connected devices, and the camera shooting condition information including at least one of the current position, the corrected position of the slave device, the shooting direction, the focal length, and the image size. to send

Preferably, the photographing direction of the slave device is parallel to the photographing direction connecting the current position and the corrected position of the slave device, and the corrected position of the slave device is determined based on the captured images.

The receiving unit 203 receives the digital content photographed using the transmitted camera photographing condition information from at least one slave apparatus among the connected apparatuses.

Also, the receiver 203 may receive head or tracking information from at least one slave device among the connected devices.

The 3D image generator 204 generates a 3D image by using the received digital content.

The server 210 includes an event information generator 211 , a transmitter 212 , a receiver 213 , and a 3D image generator 214 .

The event information generator 211 generates image capturing event information including a movement path for storing digital content and target object information.

The transmitter 212 transmits the image capturing event information to the master device, and transmits object of interest information including 3D coordinates and color information of the object of interest to the master device.

The transmitter 212 may transmit a result of calculating the current position of the master device and the corrected positions of the connected devices and camera shooting condition information to the master device.

The camera photographing condition information may include at least one of a current position of at least one slave device among the connected devices, a corrected position of the slave device, a shooting direction, a focal length, and an image size.

Preferably, the photographing direction of the focal length is parallel to the photographing direction connecting the corrected position and the target object position.

Preferably, the shooting direction of the captured images is parallel to the shooting direction connecting the position of the target object and the position of at least one of the connected devices.

The receiving unit 213 receives the captured images of the connected devices transmitted by the master device, and receives an image captured by the connected device using the camera shooting condition information.

The 3D image generator 214 generates a 3D image by using the received digital content. When generating a real-time 3D image, the master device 200 may also generate a real-time 3D image using digital content received from a plurality of connected devices and 3D information of the object of interest.

Preferably, a 3D image is generated by determining an interpolation plane between captured images having overlapping regions, and having the same frame size based on the object of interest information.

The slave device 220 includes a receiving unit 221 , a photographing unit 222 , an interface unit 223 , a display unit 224 , and a transmitting unit 225 .

The receiver 221 receives camera shooting condition information including at least one of a current corrected position of a slave device, a position of the target object, a shooting direction, a focal length, and an image size from the master device.

The current corrected position is determined in consideration of the target object position or the object of interest position in the captured image.

Preferably, the photographing direction is parallel to a line connecting the position of the target object and the currently corrected position, and the focal length is determined according to a distance between the position of the target object and the corrected current position. The resolution of the digital content may be determined according to a focal length.

The photographing unit 222 generates digital content in consideration of the received camera photographing condition information.

It is preferable to display interface content according to the user's head or eye tracking information, and record the digital content according to the displayed interface content.

The interface unit 223 collects user reaction and feedback information.

The display unit 224 outputs display information such as augmented reality (AR)/virtual reality (VR)/3D avatar/navigation information information.

The transmitter 225 transmits the captured image including the target object or the object of interest to the master device. In addition, the transmitter 225 transmits the generated digital content to the master device.

The captured image may include a marker for detecting the target object.

3 illustrates changes in a focal length and a photographing angle (AOV) when an object is photographed while maintaining the same photographing field of view (FOV).

The resolution of the image is determined by the sensor-width of the image sensor, and the angle of view of the image is calculated as in Equation (2).

Referring to FIG. 3 , in the case of photographing a distant object while maintaining the same photographing field of view, the focal length must be increased and the image must be photographed with a higher resolution. In FIG. 3 , it is an image taken at a focal length of 27 mm, and it can be seen that the sensor width is large.

For example, when the distance between the master device vehicle 100 and the mobile device is short, the distance between the mobile device and the object to be photographed is relatively long, and thus the focal length is long and the resolution is high in order to maintain the same photographing field of view. Conversely, when the distance between the mobile device and the object to be photographed is short, the focal length is relatively short and the resolution is low.

4 shows an example of a color marker used for easy recognition between connected devices.

In the case of splicing multiple images, a white balance process is required to adjust the brightness values of the images under the same shooting conditions. It is preferable to divide the white balance into color control according to the brightness of the light source and color control according to the temperature of the light source.

In order to perform white balance, main colors to be extracted include red/green/blue/yellow/magenta/cyan/gray color. Therefore, if a color matrix composed of the primary colors used for white balance is configured and the corresponding color matrix is attached to a connected device, it is possible to easily recognize the connected devices and calculate the location, and determine the shooting direction in the recognized direction. have.

Also, it is possible to perform white balance using the color marker values recognized in the image. The principle of white balance is to compare and adjust the actual R/G/B values of the primary colors present in the color marker with the R/G/B values of the primary colors in the photographed image markers.

Referring to Figs. 4(a) and 4(b), primary colors R1, R2, and R3 are indicated as reference colors of color markers, and C1 to C6 are shown as code colors.

Because multiple devices use the same color marker, a color marker cannot be used as a device's unique hardware ID. Therefore, it is desirable to reconstruct the color code value of the color marker in consideration of the hardware unique ID of the device and the camera specification of the hardware.

As shown in Fig. 4(c), multiple synthesis matrix layers are configured in the same way as the matrix dimension (4*4) of the color marker, and each composite matrix layer information is configured in consideration of the device's unique hardware ID, hardware specification, and other information. Then, a unique virtual color marker value can be generated by calculating the matrix product or the matrix sum of the color matrix value used in FIG. 4( a ).

Therefore, when the camera of the device recognizes the peripheral device, it can be recognized as a unique ID of the hardware or can be recognized as a virtual color marker value. For example, upon recognizing a color marker, devices transmit and receive information for position calculation. In this case, since the virtual color marker value is extracted using the received hardware unique ID value, the target object or the master device vehicle 100 can be recognized using the virtual color marker.

5 is a diagram illustrating a state of wireless connection between a plurality of vehicles and a plurality of mobile devices for digital content recording according to another embodiment of the present invention.

The virtual machine or hub device 520 is preferably a server connected to perform a specific event among various virtual machines. The virtual machine or hub device 520 stores and manages object of interest information including image capturing event information for storing 3D digital content, a precision map, and 3D location information (3d points) and color information related to the object of interest.

Also, the image capturing event information is transmitted to the vehicle master device 510 . It can also be set to be simultaneously transmitted to the mobile master device (530, 540). For effective network data transmission, between the vehicle master device 510 and the virtual machine 520 , the vehicle master device 510 and the mobile master device 530 , and the vehicle master device 510 and the mobile master device 540 . communication takes place.

In addition, the vehicle master device 510 transmits necessary information to the neighboring slave vehicle devices 512 , 512 , 513 , 514 , and 515 or transfers the collected information to the virtual machine 520 . The mobile master devices 530 and 540 transmit necessary information to the neighboring mobile slave devices 531 , 532 , 541 , and 542 , or transfer the received information to the vehicle master device 510 .

In this case, images are simultaneously recorded using cameras mounted on all vehicles and mobile devices, and the recorded images are transmitted to the virtual machine 520 to effectively generate 3D image data. At this time, it is preferable that the photographing direction of the cameras mounted on all mobile devices is the same as the photographing direction of the camera mounted on the vehicle master device 510 . In addition, in order to extract images that satisfy the same shooting field of view and a certain resolution, the neighboring slave vehicle devices 511 , 512 , 513 , 514 , and 515 reflect the distance difference with the vehicle master device 510 to photograph the surrounding object. An image is generated by adjusting the focal length for the image and adjusting the resolution accordingly.

The mobile master device (530, 540) and the surrounding mobile slave devices (531, 532, 541, 542) also reflect the distance from the vehicle master device (510) so that the image acquired from the mobile device can satisfy the same shooting field of view and a certain resolution It is preferable to adjust the focal length and adjust the image resolution accordingly.

The adjustment of the focal length may be calculated through images and location information captured by the neighboring slave vehicle devices 511 , 512 , 513 , 514 , and 515 transmitted to the vehicle master device 510 . When the object of interest is included in the image transmitted by the neighboring slave vehicle devices 511 , 512 , 513 , 514 , and 515 to the vehicle master device 510 , the neighboring slave vehicle devices 511 , 512 , 513 , 514 , 515 and the vehicle The master device 510 may determine to exist in the same photographing direction. Also, location information on the vehicle master device 510, the neighboring slave vehicle devices 511, 512, 513, 514, 515, and the object of interest on a network may be calculated.

At this time, the vehicle master device 510 uses its own location information and the precise road map stored in the DB, 3D location information and image color information for the object of interest, so that the accurate surrounding slave vehicle devices 511 , 512 , 513 , 514 are used. , 515) can be calculated. Accordingly, the vehicle master device 510 includes the focal length, image resolution, location of the vehicle master device 510, and the vicinity of the camera shooting condition information for image capturing of the neighboring slave vehicle devices 511, 512, 513, 514, and 515. The slave vehicle device 511 , 512 , 513 , 514 , 515 may include and provide location information.

As another embodiment of the present invention, a case in which a plurality of vehicle master devices exist will be described.

In FIG. 5 , one vehicle master device 510 is assumed, but if it is assumed that there are multiple vehicle master devices, several sub vehicle groups are set based on each vehicle master device, and each sub vehicle group is a sub vehicle group. You can set the master device.

The mobile device group may also group mobile devices as many as the number of sub mobile groups. In this case, by selecting a sub mobile group master device of each sub mobile group, it is possible to effectively control transmission and reception of group data.

Each sub mobile group (including the sub mobile group and the sub vehicle group) transmits camera shooting condition information to the sub vehicle group master device so that an image can be taken based on the shooting direction of the sub vehicle group master device or the sub mobile group master device Or it is preferable to receive from the sub mobile group master device.

As another embodiment according to the present invention, the virtual machine or the hub device 520 may transmit and receive camera shooting condition information to and from each sub mobile group.

In FIG. 5 , the vehicle/mobile group transmits/receives camera shooting condition information through the vehicle master device 510/mobile master device 530 and 540 . However, without selecting the vehicle master device 510 , transmission and reception of camera shooting condition information required for all devices is possible through the virtual machine or the hub device 520 .

In this case, the virtual machine or hub device 520 provides camera shooting condition information for calculating the location of devices connected to the network and capturing images of the devices.

In addition, in order to produce a 3D image, the image must be captured based on the shooting direction of the target object or the target device. Therefore, the virtual machine or hub device 520 groups the vehicle or mobile device based on the target object or target device, and determines the shooting direction of the target object or target device using the target object location or target device location and marker information for each group. The reflected image is provided to the virtual machine or the hub device 520 . The virtual machine or hub device 520 may calculate locations of vehicles or mobile devices using the received image and object of interest information.

Therefore, the virtual machine or hub device 520 reflects the distance between the target object or the target device and the vehicle or mobile devices included in the same group, and includes information on the focal length and resolution for capturing images of the vehicle or mobile devices. Information on shooting conditions can be provided.

6 illustrates an example of stitching a plurality of images in order to process a captured image into a 3D image.

Referring to FIG. 6 , images having the same photographing field of view in each photographed image are classified into groups. Images having overlapping areas are arranged in the form of a panoramic image by using focus coordinates and shooting directions for each classified group. The images arranged in this way are joined to create a tile image.

It is also possible to rejoin multiple tile images by using the virtual camera focus to join the tile images. The virtual camera focus used at this time is a travel path of the vehicle master device 410, the target device, or the target object, or the sub vehicle group master device, the sub mobile group master device, the sub target device, or the sub target object. may be a movement path of

Overlapping images with overlapping regions are spliced using the same 3D coordinates for the same object of interest. However, since a parallax occurs between images taken based on a plurality of focal points, the same object of interest does not exist on the same image plane in a stereo image. Therefore, it is necessary to form a fitting plane between stereo images by using 3D coordinates existing in the overlapping image region between adjacent images. In this case, it is preferable to select the interpolation plane as a plane in which the adjustment value of the used 3D coordinate value is minimized.

7 is a flowchart of a method of recording digital content in a slave device among connected devices according to an exemplary embodiment of the present invention.

Referring to FIG. 7 , the method for recording digital content according to the present embodiment includes steps that are time-series processed by the slave device for recording digital content shown in FIG. 2 . Therefore, even if omitted below, the descriptions of the slave device for recording digital content shown in FIG. 2 are also applied to the method for recording digital content according to the present embodiment.

7 is a diagram illustrating a process of generating image information based on a photographing direction of a master device or a target object in a slave device among connected devices.

In step 710 , the slave device for recording digital content receives image capturing event information from the master device 200 . The image capturing event may be received from a server such as a hub device.

For efficient network communication, communication is controlled centered on the master device 200, and after dividing the peripheral connected devices into sub-groups, the sub-master device of each sub-group is selected to control communication with the surrounding mobile devices. have.

In step 720, the slave device for recording digital content detects the master device by checking the physical marker. Since a 3D image production is possible only when a field of view is secured between the slave device and the master device, in order to effectively recognize an image between the devices, it is preferable to recognize the marker information present on the device and then proceed with image creation.

In step 730, the slave device for recording digital content transmits the front/rear/side images taken by reflecting the location information of the master device to the master device.

In step 740, the master device uses the front/rear/side images taken by the slave device to the position of the master device, 3D coordinate information included in the DB-stored object of interest information, color information of the image, and the precision road map to the slave device. Calculate the location information of the device.

The coordinates of the slave device corrected by the triangulation method may be calculated by referring to the positions of the master device and the respective devices of the slave device. The position calculation of the slave device and the transmission of camera shooting condition information for image shooting may be performed by the master device, the sub-master device, or the hub device according to the system configuration.

Also, the master device may calculate distance information between the master device and the slave device using the calculated slave device position coordinates. The master device may provide camera shooting condition information including focal length information and image size required for slave devices to generate images having the same shooting field and frame size to the slave device. The size of the image is determined in consideration of the distance between the master device and the slave device, and the master device may divide a plurality of connected devices into digital content generation groups and set each group to have the same image size.

Information for calculating a focal length and a resolution of a photographed image of the camera photographing condition information may be performed by a slave device. In the slave device, the focal length and the photographed image resolution may be calculated by using the master device location and the slave device location and distance information between the photographing target and the photographing target.

In step 750, the slave device for recording digital content receives corrected position information and camera shooting condition information of the slave device from the master device or the sub-master device.

In step 760, the slave device for recording digital content determines the shooting direction by recognizing the marker attached to the master device.

Therefore, based on the location information of the master device recognized through the marker, the effective image taking direction angle can be automatically changed in the slave device.

In step 770, when the user determines the camera shooting direction in consideration of the location of the master device, the slave device for recording digital content uses the recognized marker information of the master device to display the preferred camera shooting direction for the user on the display. can do.

In addition, the slave device for recording digital content may acquire the user's head/eye movement information. In order to effectively collect user reaction and feedback information, the slave device may monitor the user's head or eye movement information. In order to effectively track the head and eyes, it is possible to effectively track eye movements by measuring glints caused by changes in the movement of the iris of the eyes using a method of analyzing images of head or eye movements and a light source.

In addition, the information displayed on the slave device may be augmented reality (AR)/virtual reality (VR)/navigation information for effective user recognition, and the master device or the main object of interest is provided in the form of a specific 3D object and avatar to be displayed. can

In step 780, the slave device for recording digital content generates digital content in the determined shooting direction.

In step 790 , the slave device for recording digital content transmits the captured digital content to the master device.

In this case, referring to FIG. 5 , it may be transmitted to the vehicle master device 510 via the mobile master devices 530 and 540 . Transmission and reception of such data is transmitted to the virtual machine or the hub device 520 through the master device or the sub-master device according to the system configuration. Also, it may be transmitted directly to the virtual machine or the hub device 520 without going through an intermediate network node.

8 is a flowchart of a method of generating a 3D image in a master device among connected devices according to an embodiment of the present invention.

Referring to FIG. 8 , the method for generating a 3D image according to the present embodiment includes steps that are time-series processed by the master device for generating the 3D image shown in FIG. 2 . Therefore, even if omitted below, the description above with respect to the master device for generating the 3D image shown in FIG. 2 is also applied to the method for generating the 3D image according to the present embodiment.

In step 810 , the master device that generates the 3D image receives image capturing event information from the server 210 .

The received image capturing event information is retransmitted to a neighboring slave vehicle device connected to a network or a neighboring mobile slave device. In addition, it is possible to receive from the server 210 a precise map of the surrounding road for location correction, 3D coordinate information of the main object of interest, color information, and reference landmark information, along with the image shooting event information. Meanwhile, the master device may be a vehicle or a mobile device, and the vehicle master device may be replaced with a mobile device having a communication function.

On the other hand, the master device 200 for generating a 3D image is connected to a neighboring slave vehicle device or a neighboring mobile slave device through a network. In this case, all connected slave devices must be capable of capturing images in the same direction as the camera of the master device 200 .

Recognition between devices can be effectively performed by recognizing the marker information displayed on the master device by using the image technique in the captured image. Therefore, only a device connected to the master device 200 over the network and recognizing the marker of the master device 200 may perform an additional image capturing step.

In step 820, the master device that generates the 3D image transmits the received image capturing event information to the slave device.

In operation 830 , the master device generating the 3D image receives at least one image from the front, rear, and side images from the slave device.

In step 840, the master device for generating the 3D image calculates the current location of the master device by comparing the collected image information and the 3D coordinate information collected using the lidar and reference landmark information.

In addition, the master device receives an image captured by recognizing the vehicle master device in the slave device received in step 830, and also calculates the position of the slave device.

It is preferable to correct the position of the slave device based on the image captured by the master device or the image captured by the slave device. In particular, recognition between the plurality of slave devices and the master device can be recognized through the marker, and the absolute direction angle of the slave device recognized as the marker can be determined based on the location information of the master device.

Step 840 may be performed by the master device 200 or the server 210 according to the system configuration. When the server 210 determines the location and generates the camera shooting condition information, the master device or the slave vehicle device needs to transmit the collected image information and Lidar information to the server 210 in real time.

In step 850, the master device that generates the 3D image transmits the corrected position information of the slave device and the camera shooting condition information. The transmission of the corresponding information may be transmitted to all slave devices or may be transmitted only to a specific mobile master device.

Camera shooting condition information, including focal length and image resolution information, location information of each slave device, and location information of the master device, is provided to each slave device so that the slave devices can shoot images having the same shooting field and frame size. It is preferable to transmit

The focal length and image resolution information of the camera shooting condition information can be calculated even in the slave device using the location information of the master device and the slave device, and the focal length information is information determined according to the distance difference between the master device and the slave device.

As the distance difference is small, the distance from the object to be photographed is relatively increased and the focal length is increased, so it is necessary to generate a high-resolution image. Conversely, as the distance difference increases, the distance between the object to be photographed and the slave device becomes relatively close, so it is effective to generate a low-resolution image with a short focal length.

In step 860, the master device that generates the 3D image transmits display information to the slave device. The master device may send various display information in order to collect the location of the slave device and the response and feedback information of the slave device user. Such display information may include augmented reality (AR)/virtual reality (VR)/3D avatar/navigation information information and the like. In addition, by collecting reaction information of a slave device user, information necessary for a specific slave device can be selectively transmitted. Head or eye tracking results may be collected from the slave device, and additional information corresponding to the analyzed result may be transmitted to a specific slave device.

In step 870, the master device that generates the 3D image receives image information from the slave device. Referring to FIG. 5 , it may be transmitted directly to a virtual machine or a hub device 520 without going through the vehicle master device 510 .

In step 880, the master device for generating the 3D image sets the virtual camera shooting direction.

In step 890, the master device that generates the 3D image generates a 3D image by using the received image.

9 is a flowchart of a method of generating a 3D image in a server among connected devices according to an exemplary embodiment of the present invention.

Referring to FIG. 9 , the method for generating a 3D image according to the present exemplary embodiment includes steps that are time-series processed by the server for generating the 3D image shown in FIG. 2 . Therefore, even if omitted below, the description above with respect to the server for generating the 3D image shown in FIG. 2 is also applied to the method for generating the 3D image according to the present embodiment.

Referring to FIG. 9 , a process in which the server 210 processes image capturing event information and generates a 3D image is illustrated.

In step 910 , the server 210 transmits image capturing event information and interest object information for acquiring an image image to the master device 200 . The image capturing event information may include a travel path for capturing an image.

The server 210 divides the connected devices into several device groups, and determines a master device in each device group. When the configuration system is composed of a plurality of servers, a specific server may be selected as the hub device 520 .

In step 920 , the master device 200 transmits the image capturing event information to the slave device 220 .

In step 930 , the master device 200 receives at least one image from the front, rear, or side images from the slave device 220 .

In step 940, the server 210 receives at least one image of the front, rear, or side from the master device.

In order for the server 210 to effectively recognize the master device and the slave device by analyzing images captured by each device in a specific device group, all devices may include physical marker information. Devices designated in the same group determine the location of the master device and acquire images in the shooting direction of the master device.

In step 950, the server 210 calculates the current position of the master device and the corrected position of the slave device. The server 210 may determine the location information of all devices by using the received image information and the object of interest information, such as a precision road map.

In step 960, the server 210, using the location information of the master device and the slave device, the camera shooting condition information including the focal length and image resolution information of the image image to be acquired from each device and the corrected position of the slave device Send to the master device.

In step 970, the master device 200 retransmits the received camera shooting condition information and the corrected position of the slave device to the group to which the master device belongs.

In step 975 , the server 210 transmits information displayed on a specific device group including slave devices. It is a function to provide additional display information to devices belonging to a selected specific device group by analyzing the user's response and feedback signals. The display information may be provided as augmented reality (AR)/virtual reality (VR)/3D avatar/navigation information, and the like.

In step 980 , the server 210 may receive information on the captured image from each slave device included in the device group to which the master device belongs, without going through the master device.

In step 990, the server 210 selects a virtual camera shooting direction to join the tile images. In this case, the location of the master device and the location of the sub-master device may be selected as the virtual camera shooting direction.

In step 995, the server 210 rejoins different tile image groups into one 3D image using the selected virtual camera shooting direction. The 3D image may be created by creating a fitting plane between images having overlap in the panoramic image. When the bonding surface is configured, image distortion occurs in the same object due to parallax occurring at different focal points in a stereo image. Therefore, an interpolation plane is determined by using 3D coordinates for an object existing in the overlapping area.

In step 995, splicing is required between images having the same photographing field of view and frame size. Such a tile image group is an image taken under similar or identical conditions when taking an image, and is divided into vehicle group images and mobile group images, and bonding is preferably performed within the same image group. In addition, it is preferable to generate a 3D image by splicing images having the same image size. The 3D image generation is also possible in the master device.

According to another embodiment of the present invention, the server 210 may directly transmit the location determination of each connected device and the transmission of camera shooting condition information to the slave device 220 instead of the master device transmitting the information.

In this case, the server 210 needs information on the target object. A specific target object is networked and contains a physical marker. Accordingly, when the mobile device can be recognized by the physical marker information, a plurality of devices can recognize the mobile device as a target object. A plurality of devices that have recognized the target object transmit images captured based on the target object to the server 210 .

The server 210 that has received the image may calculate the position values of the target object and the slave device by using the received image and the object of interest information, and take an image with the slave device according to the distance difference between the target object and the slave device. It is possible to transmit camera shooting condition information for

In this case, the camera shooting condition information includes a target object position, a slave device position, and focal length and image resolution information according to a distance difference between the target object and the slave device.

The focal length and the image resolution may be calculated not only in the server 210 but also in the slave device. Preferably, the slave device is an automated device capable of automatically recognizing a target object through marker recognition and automatically determining an image capturing angle based on the target object position. When the slave device determines the image capturing angle by the user, it is necessary to provide display information for the effective image capturing by the user. Such display information may be selectively provided to the slave device according to the photographing state of the slave device.

In addition, augmented reality (AR)/virtual reality (VR)/3D avatar/navigation information may be provided as display information to effectively process the user's reaction and feedback signals.

In order to provide effective display information, the slave device may track the user's head or eye movements. The tracking information may be generated by analyzing a head or eye movement using an image analysis technique, or detecting a change in glint according to an iris movement of the eye using a light source to recognize the eye movement. Additional display information may be provided based on the tracking information.

In addition, the head or eye movement is provided to the server 210 , and the server 210 may selectively provide effective display information to a specific slave device using the received head or eye movement information.

Embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The term '~ unit' used in this embodiment means software or hardware components such as field-programmable gate array (FPGA) or ASIC, and '~ unit' performs certain roles. However, '-part' is not limited to software or hardware. The '~ unit' may be configured to reside on an addressable storage medium or may be configured to refresh one or more processors. Thus, as an example, '~' denotes components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays, and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

All of the above-described functions may be performed by a processor such as a microprocessor, a controller, a microcontroller, an application specific integrated circuit (ASIC), etc. according to software or program code coded to perform the functions. The design, development and implementation of the above code will be apparent to those skilled in the art based on the description of the present invention.

Although the present invention has been described with reference to the embodiments, those of ordinary skill in the art can variously modify and change the present invention within the scope without departing from the technical spirit and scope of the present invention. You will understand. Therefore, the present invention is not limited to the above-described embodiments, and it will be said that all embodiments within the scope of the following claims are included.

Claims (17)

transmitting a captured image including the same target object or object of interest from at least one or more of the plurality of connected devices to one of the plurality of connected devices;
receiving camera shooting condition information including at least one of a corrected position of the current device and a position of the target object, a shooting direction, an image size, and a focal length from one of the plurality of connected devices;
generating digital content in consideration of the received camera shooting condition information; and
transmitting the generated digital content to one of the plurality of connected devices;
The corrected position of the current device is determined in consideration of the target object position or the object of interest position in the captured image,
The shooting direction is parallel to a line connecting the position of the target object and the corrected position of the current device, and the focal length is determined according to a distance between the position of the target object and the corrected current position,
The at least one or more of the plurality of connected devices generates digital content with an image size included in the shooting condition information. According to claim 1,
The method of claim 1, wherein the captured image includes a marker that detects the target object. According to claim 1,
The method of claim 1, wherein the resolution of the digital content is determined according to the focal length. According to claim 1,
The digital content generating method, characterized in that it includes the user's head or eye tracking information. determining a current location using the captured image of the object of interest;
Camera shooting condition information including at least one of the current position and the corrected position, shooting direction, image size, and focal length of the one or more connected devices to at least one connected device among a plurality of connected devices transmitting; and
Receiving the digital content photographed in the same image size by using the transmitted camera shooting condition information from at least one or more connected devices among the plurality of connected devices,
the corrected position is determined based on the captured images;
The photographing direction is a digital content creation method, characterized in that parallel to an extension line connecting the current position and the corrected position. 6. The method of claim 5,
The digital content creation method according to claim 1, wherein the plurality of connected devices are classified into groups to receive digital content having different image sizes. 6. The method of claim 5,
and detecting markers attached to the plurality of connected devices in the captured image. 6. The method of claim 5,
and receiving head or eye tracking information from at least one of the connected devices. transmitting image capturing event information including information on a moving path and target object for storing digital content to a master device;
transmitting object of interest information including 3D coordinates and color information of the object of interest to the master device;
calculating the current position of the master device and the corrected positions of the connected devices, and transmitting the calculation result and camera shooting condition information to the master device;
receiving, through the master device, a captured image captured by the connected devices using the camera shooting condition information; and
generating a 3D image by using the received digital content;
The capturing direction of the captured images is parallel to an extension line connecting the position of the target object and the position of at least one of the connected devices. 10. The method of claim 9,
The camera shooting condition information includes focal length, image size, corrected positions of the connected devices, and the target object position,
3D image generating method of a content server, characterized in that the image resolution is determined based on the focal length information. 10. The method of claim 9,
3D image generating method of a content server, characterized in that the received captured image is classified into different groups according to the image size included in the camera shooting condition information, and then a 3D image is generated. 10. The method of claim 9,
For the captured images having the same image size included in the camera shooting condition information, determining an interpolation plane between captured images so that a change in coordinate information of an object of interest existing in an overlapping area is minimized A method of generating 3D images in a content server characterized by the 10. The method of claim 9,
The content server 3D image generation method further comprising the step of determining a position of a virtual camera connecting the captured images. a transmitter configured to transmit a captured image including the same target object or object of interest from at least one or more connected devices to one of the plurality of connected devices;
a receiver configured to receive camera shooting condition information including at least one of a corrected position of the current device, a position of the target object, a shooting direction, a focal length, and an image size from one of the plurality of connected devices; and
and a photographing unit that generates digital content in consideration of the received camera photographing condition information,
The transmitter transmits the generated digital content to one of the plurality of connected devices, and the currently corrected position is determined in consideration of a target object position or an object of interest position in the captured image, and the shooting direction is parallel to a line connecting the position of the target object and the current corrected position, and the focal length is determined according to a distance between the position of the target object and the corrected current position. a calculator for determining a current location using the captured image of the object of interest;
Transmitting camera shooting condition information including at least one of the current position, the corrected position of the one or more connected devices, a shooting direction, a focal length, or an image size to at least one or more connected devices among a plurality of connected devices a transmitter to;
a receiver configured to receive digital content generated by using the camera shooting condition information from at least one connected device among the plurality of connected devices; and
The current location is determined by comparing a boundary line or color between the object of interest information and the captured image, and the photographing direction is parallel to an extension line connecting the current location and the corrected location of the one or more connected devices; and the calibrated position of the one or more connected devices is determined based on the captured images. an event information generating unit generating image capturing event information including a movement path and target object information for storing digital content;
Transmits the image capturing event information to the master device, transmits object of interest information including 3D coordinates and color information of the object of interest to the master device, and calculates the current position of the master device and corrected positions of connected devices and a transmitter for transmitting the calculation result and camera shooting condition information;
a receiver configured to receive captured images generated using camera shooting condition information in the connected devices through the master device; and
and a 3D image generator configured to generate a 3D image by grouping the received digital content by digital content by using the image size information included in the shooting information,
The capturing direction of the captured images is parallel to an extension line connecting the position of the target object and the position of at least one of the connected devices. A computer-readable recording medium in which a program for executing the method of any one of claims 1 to 13 on a computer is recorded.

Images