KR102407190B1 - Image capture apparatus and method for operating the image capture apparatus - Google Patents

Abstract

An image photographing apparatus and an operating method of the image photographing apparatus are disclosed according to embodiments. An image photographing apparatus includes: a photographing unit photographing an image; and a control unit connected to the photographing unit, wherein the control unit obtains a background image having depth information, arranges a 3D virtual image used to represent an object in the background image, and adds the 3D virtual image to the arranged 3D virtual image. Controls the photographing unit to photograph the object based on the photographing unit.

Description

Image capture apparatus and method for operating the same

Various embodiments relate to an image capturing apparatus and an operating method thereof.

In recent years, electronic cameras have been widely used. In an electronic camera, an image of a subject is formed on a semiconductor imaging device, for example, an image sensor through an imaging optical system. This object image is converted into an electric signal, and the image data of the still image thus obtained is recorded on a recording medium such as a semiconductor memory or a magnetic disk.

As the popularity of photographing devices (eg, cameras, smart phones, tablet computers, smart wearable devices) increases, people are not satisfied with simply photographing postures, and according to different times, different places, different scenarios (eg, places Shooting using )) I hope to have a complicated shooting posture or an interesting joongyeong posture.

1 shows an example of photographing using a position.

101 of FIG. 1 illustrates an object photographing posture using the position of the sun. 102 of FIG. 1 exemplifies the posture of the close-up object using the location of the remote object.

However, it is generally difficult to photograph a complex posture or an interesting posture through only a photographing object. Although it is possible to assist the shooting object by adjusting the shooting posture, it is difficult to guarantee the shooting effect, and the user experience is poor. Moreover, it is difficult for the shooting object and/or the photographer to know in advance the final shooting effect for effective adjustment.

Various embodiments are directed to providing a photographing apparatus and method for effectively photographing a complex photographing posture or an interesting photographing posture of an object.

An image photographing apparatus according to an embodiment includes: a photographing unit photographing an image; and a control unit connected to the photographing unit, wherein the control unit obtains a background image having depth information, arranges a 3D virtual image used to represent an object in the background image, and adds the 3D virtual image to the arranged 3D virtual image. Controls the photographing unit to photograph the object based on the photographing unit.

According to an embodiment, the controller may determine a 3D virtual space position and/or a 3D posture of the 3D virtual image.

According to an exemplary embodiment, the controller may determine the photographing accessory and further configure a three-dimensional spatial position and/or a three-dimensional posture of the photographing accessory in the background image.

According to an embodiment, the controller is configured to arrange the three-dimensional virtual image based on the shooting appendage of the three-dimensional virtual image in the background image, the three-dimensional virtual image and the three-dimensional virtual image The relationship between the shooting accessories may be defined by a pre-configured shooting scenario template.

According to an embodiment, the controller may configure a three-dimensional posture of the three-dimensional virtual image by arranging a three-dimensional space position of a joint point of the three-dimensional virtual image.

According to an embodiment, the controller may determine a difference between the object and the 3D virtual image, and output a photographing guide through the photographing apparatus and/or a wearable device based on the determined difference .

According to an embodiment, the control unit detects the object based on the 3D posture of the 3D virtual image, detects the object based on 3D spatial location information provided by the wearable device, or detects the object based on the object selected by the user. It can be detected based on the target tracking (target tracking).

According to an embodiment, the difference between the object and the 3D virtual image may include a 3D spatial position difference and/or a 3D posture difference between the object and the 3D virtual image.

According to an embodiment, the controller is configured to determine the 3D posture difference between the object and the 3D virtual image in a 3D space between a 3D spatial position of a bonding point of the object and a corresponding bonding point of the 3D virtual image. It can be determined by comparing the positions.

According to an embodiment, the controller detects the posture of the object in the background image, determines whether the posture of the object matches the focusing posture, and when the detected posture of the object matches the focusing posture, the object The photographing unit may be controlled to focus and photograph.

According to an embodiment, the focusing posture may be a pre-configured posture used to indicate a desired photographing posture of the photographing object, or may be any photographing posture in a pre-configured photographing posture database.

According to an embodiment, the control unit determines an expected posture of the three-dimensional virtual image used to predict the appearance of the posture of the object desired to be captured, and automatically sets the object when detecting that the object assumes the posture. It is possible to control the photographing unit to focus and capture.

According to an embodiment, the control unit determines multiple postures of the three-dimensional virtual image used to represent a series of object postures desired to be continuously photographed, and the three-dimensional virtual image in the background image for a preset period of time when the object is set in advance and continuously matches, the photographing unit may be controlled to automatically focus the object and continuously photograph the object.

According to an embodiment, the object includes multiple objects, and the controller may remove an object that does not match the 3D virtual image before or after the object is focused and photographed.

According to an exemplary embodiment, a method of operating a photographing apparatus includes an operation of acquiring a background image having depth information, an operation of arranging a 3D virtual image used to represent an object on the background image, and an operation of placing the 3D virtual image on the arranged 3D virtual image. and capturing the object based on the image.

A computer-readable recording medium in which a program for executing a photographing method is recorded according to an embodiment is disclosed.

According to various embodiments, a user may effectively photograph a complex or interesting photographing posture of an object.

1 shows an example of photographing using a location.
2 is a schematic block diagram of a photographing apparatus 100 according to an exemplary embodiment.
3A is a detailed block diagram of a configuration of a photographing apparatus according to an exemplary embodiment.
FIG. 3B is a block diagram of a module that may be stored in the program storage unit 170 shown in FIG. 3A to perform a method of operating a photographing apparatus according to an exemplary embodiment.
4 is a flowchart illustrating a photographing method according to an exemplary embodiment.
5 illustrates a three-dimensional virtual image according to an embodiment.
6 illustrates an example of selecting a 3D virtual image according to an exemplary embodiment.
7 illustrates another example of selecting a 3D virtual image according to an exemplary embodiment.
8 and 9 show examples of notifying a user of a location area according to an embodiment.
10 illustrates an example of automatically configuring a 3D spatial position of a 3D virtual image according to an exemplary embodiment.
11 illustrates an example of configuring a 3D spatial position of a 3D virtual image according to an exemplary embodiment .
12 illustrates an example of configuring a rotation angle of a 3D virtual image according to an embodiment.
13 illustrates an example of selecting a joint point of a 3D virtual image according to an exemplary embodiment.
14 illustrates an example of configuring a 3D spatial position of a joining point of a 3D virtual image according to an embodiment.
15 illustrates an example of informing a user that a three-dimensional spatial location of a joining point is incorrectly configured, according to an embodiment.
16 illustrates an example of selecting and configuring a 3D virtual image and a joint point of the 3D virtual image together, according to an embodiment.
17 is a flowchart illustrating a method of performing imaging based on a 3D virtual image constructed according to an exemplary embodiment.
18 illustrates an example of guiding a photographing object to reach a designated position using an image guide, according to an embodiment.
19 illustrates an example of guiding a plurality of photographing objects to reach designated positions, according to an embodiment.
20 illustrates an example of guiding a photographing object to adjust a posture of the photographing object, according to an exemplary embodiment.
21 illustrates an example of guiding a plurality of photographing objects in order to adjust postures of the plurality of photographing objects according to an exemplary embodiment.
22 illustrates an example of a connected electronic device according to an embodiment.
23 illustrates another example of displaying on a connected electronic device, according to an embodiment.
24 is a flowchart illustrating a method of performing imaging based on a configured 3D virtual image, according to another exemplary embodiment.
25 illustrates an example of focusing each of a plurality of objects according to an exemplary embodiment.
26 illustrates another example of focusing each of a plurality of photographing objects according to an exemplary embodiment.
27 illustrates an example of capturing a photographing object according to an exemplary embodiment.
28 illustrates an example of recognizing a target object and a non-target object according to an embodiment.
29 illustrates an example of removing a non-target object according to an embodiment.
30 illustrates an example of selecting an imaging accessory and configuring a three-dimensional spatial position of the imaging accessory according to an embodiment.
31 illustrates an example of configuring a three-dimensional spatial position of an imaging accessory according to an embodiment.
32 illustrates an example of configuring a rotation angle of an imaging accessory according to an embodiment.
33 is a flowchart illustrating a process of a photographing method according to an embodiment.
34 is a flowchart illustrating a photographing method according to another exemplary embodiment.
35 illustrates an example of determining whether a posture of a photographing object matches a focusing posture, according to an exemplary embodiment.
36 illustrates another example of capturing a photographing object according to an embodiment.
37 illustrates another example of removing a non-target object according to an embodiment.
38 is a flowchart illustrating a photographing method according to an exemplary embodiment. This method is used for a single shooting object.
39 is a flowchart illustrating a photographing method according to an exemplary embodiment.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

In the entire specification, when a part <includes> a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as <... unit> and <module> described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

2 to 39 , photographing methods according to various embodiments will be described. The photographing methods may be implemented through a photographing device or may be implemented through a computer program. For example, the method may be implemented through a camera installed in the photographing apparatus, or may be implemented through a function program in an operating system of the photographing apparatus.

2 is a schematic block diagram of a photographing apparatus 100 according to an exemplary embodiment.

The photographing device 100 according to an embodiment is, for example, a camera, a mobile communication terminal (eg, a smartphone), a smart wearable device (eg, a smart watch, a smart bracelet, smart glasses, etc.), a personal computer, a tablet computer, such as, It may be an electronic terminal having a photographing function. A module included in the photographing device may be implemented together with a specific component (eg, a sensor). As an example, the module may be implemented in a digital signal processor, a general-purpose hardware processor such as a field programmable gate array (FPGA), or a dedicated hardware processor such as a dedicated chip, or software such as a computer program. For example, the computer program is implemented as a module installed in the camera application of the photographing device, or is implemented as a function program in the photographing device operating system.

Referring to FIG. 2 , the photographing apparatus 100 may include a photographing unit 110 , a control unit 120 , and a display unit 130 .

The photographing unit 110 may include at least one camera, and may include an infrared (IR) camera. Also, when the optical system moves, the photographing unit 110 may move in the same direction.

The display 130 may display at least one object. The display unit 130 may display a viewfinder image so that a composition of a subject to be photographed and a photographing condition can be checked.

In addition, the display unit 130 may be formed of a liquid crystal display panel (LCD), an organic light emitting display panel, or the like.

According to an embodiment, the display unit 130 may be implemented as a touch sensitive display capable of receiving a user input.

The controller 120 controls components of the photographing apparatus 100 such as the photographing unit 110 and the display unit 130 .

According to an embodiment, the controller 120 acquires a background image having depth information, arranges a 3D virtual image used to represent an object on the background image, and captures the object based on the arranged 3D virtual image. The photographing unit 110 may be controlled.

According to an embodiment, the controller 120 may determine a 3D virtual space location and/or a 3D posture of the 3D virtual image.

According to an exemplary embodiment, the controller 120 may determine a photographing accessory and further configure a three-dimensional spatial position and/or a three-dimensional posture of the photographing accessory in the background image.

According to an embodiment, the controller 120 is configured to arrange the three-dimensional virtual image based on the shooting appendage of the three-dimensional virtual image in the background image, the three-dimensional virtual image and the three-dimensional virtual image A relationship between the shooting adjuncts of may be defined by a pre-configured shooting scenario template.

According to an embodiment, the controller 120 may configure the 3D posture of the 3D virtual image by arranging 3D spatial positions of joint points of the 3D virtual image.

According to an embodiment, the controller 120 may determine a difference between the object and the 3D virtual image, and output a photographing guide through the photographing apparatus and/or a wearable device based on the determined difference .

According to an embodiment, the controller 120 detects the object based on the 3D posture of the 3D virtual image, detects the object based on 3D spatial location information provided by the wearable device, or detects the object based on the object selected by the user. It can be detected based on the target tracking (target tracking).

According to an embodiment, the controller 120 is configured to determine the three-dimensional posture difference between the object and the three-dimensional virtual image, a three-dimensional space position of a joint point of the object and a corresponding joint point of the three-dimensional virtual image. It can be determined by comparing spatial positions.

According to an embodiment, the controller 120 detects the posture of the object in the background image, determines whether the posture of the object matches the focusing posture, and when the detected posture of the object matches the focusing posture, the object The photographing unit 110 may be controlled to focus and photograph.

According to an embodiment, the controller 120 determines an expected posture of the three-dimensional virtual image used to predict the appearance of the posture of the object desired to be captured, and automatically sets the object when detecting that the object assumes the posture. It is possible to control the photographing unit to focus and capture.

According to an embodiment, the controller 120 determines multiple postures of the three-dimensional virtual image used to represent a series of object postures desired to be continuously photographed, and the three-dimensional virtual image in the background image for a preset period of time when the object is set in advance. and continuously matches, the photographing unit may be controlled to automatically focus the object and continuously photograph the object.

According to an embodiment, the controller 120 may remove an object that does not match the 3D virtual image before or after the object is focused and photographed.

3A is a detailed block diagram of a configuration of a photographing apparatus according to an exemplary embodiment.

Referring to FIG. 3A , the photographing device 100 includes a photographing unit 110, a control unit 120, a display unit 130, a display driving unit 131, an analog signal processing unit 140, an image signal processing unit 141, a memory 150, a storage/reading control unit 160, a memory card 161, and a program storage. It includes a unit 170, an operation unit 180, and a communication unit 190.

The photographing unit 110 is a component that generates an image of an electrical signal from incident light, and includes a photographing lens 1, an iris 2, a focus lens 3, a light splitter 4, a shutter 5, an image sensor 6, an image sensor controller 7, an AF device 8, and a focus. It includes a lens driving unit 9, a lens driving unit 10, and a focal length detecting unit 11.

The photographing lens 1 may include a plurality of groups and a plurality of lenses for photographing a subject. The position of the lens 1 is adjusted by the lens driving unit 10 . The lens driver 10 adjusts the position of the lens 1 according to the control signal provided from the controller 120 .

The opening and closing degree of the diaphragm 2 is controlled by the diaphragm driver (not shown), and the amount of light incident to the image sensor 6 is adjusted.

The position of the focus lens 3 is adjusted by the focus lens driver 9 and focusing is performed.

The light splitter 4 divides the light beam that has passed through the photographing lens and the focusing lens into a photographing light beam and a light beam toward the AF (autofocus) system. For example, the light splitter 4 may be a half-mirror. One of the light beams divided by the light splitter 4 is incident on the image sensor 6 .

The optical signal transmitted through the light splitter 4 reaches the light receiving surface of the image sensor 6 to form an image of the subject. The imaging sensor 6 may be a Charge Coupled Device (CCD) image sensor or a Complementary Metal Oxide Semiconductor Image Sensor (CIS) that converts an optical signal into an electrical signal. The imaging sensor controller 7 temporally limits the amount of light to the imaging sensor 6 by controlling the shutter 5 included in the imaging sensor 6 . When data is read from the image sensor 6, the shutter 5 is closed. The imaging sensor controller 7 may control the imaging sensor 6 according to a control signal automatically generated by an image signal input in real time or a control signal manually input by a user's manipulation.

The phase difference AF device 8 represents a phase difference type distance measuring device that functions as a focus detection device that measures the distance to the subject after receiving the light passing through the light splitter 4 . The phase difference AF device 8 may include an AF mirror for changing the direction of the AF optical axis, a separator lens for dividing the pupil of the AF light beam, and an AF sensor for performing phase difference type distance specification (phase difference AF).

The output from the AF device 8 is input to the control unit 120 and used by the AF control unit 121 for AF control of phase difference AF. The AF control unit 121 in the control unit 120 may also perform AF control of contrast AF based on the output from the image pickup device.

The analog signal processing unit 140 performs noise reduction processing, gain adjustment, waveform shaping, analog-to-digital conversion processing, and the like on the analog signal supplied from the imaging sensor 6 .

The image signal processing unit 141 is a signal processing unit for processing a special function on the image data signal processed by the analog signal processing unit 140 . For example, noise reduction for input image data, gamma correction, color filter array interpolation, color matrix, color correction, color enhancement ( color enhancement) Image signal processing for improving image quality such as white balance adjustment, luminance smoothing, and color shading and providing special effects may be performed. The image signal processing unit 141 may compress input image data to generate an image file, or may restore image data from the image file.

The image data output from the image signal processing unit 141 is input to the storage/reading control unit 160 through the memory 150 or directly, and the storage/reading control unit 160 stores the image data in the memory card 161 according to a signal from a user or automatically. In addition, the storage/reading control unit 160 may read image data from an image file stored in the memory card 161 and input it to the display driver 131 through the memory 150 or another path to display the image on the display unit 130. . The memory card 161 may be removable or permanently installed in the photographing device 100 . For example, the memory card 161 may be a flash memory card such as an SD (Secure Digital) card.

In addition, the image signal processing unit 141 may perform display image signal processing for display on the display unit 130 . For example, luminance level adjustment, color correction, contrast adjustment, outline enhancement adjustment, screen division processing, character image generation, etc., image synthesis processing, etc. can be performed.

The image signal processing unit 141 according to an embodiment may detect a target object by performing face recognition as image recognition processing from the input image data.

Meanwhile, the signal processed by the image signal processing unit 141 may be input to the control unit 120 through the memory 150 or may be input to the control unit 120 without going through the memory 150. Here, the memory 150 operates as a main memory of the photographing apparatus 100, and temporarily stores necessary information while the image signal processing unit 141 or the control unit 120 operates. The program storage unit 170 may store programs such as an operating system and an application system for driving the photographing device 100 .

The photographing device 100 includes a display unit 130 to display an operation state of the photographing device 100 or image information captured by the photographing apparatus 100 . The display unit 130 may provide visual information to the user. In order to provide visual information, the display unit 130 may be formed of, for example, a liquid crystal display panel (LCD), an organic light emitting display panel, or the like.

According to an embodiment, the display unit 130 may be a touch screen capable of recognizing a touch input. Accordingly, the photographing device 100 may display a menu such as virtual image selection through the display unit 130 and receive a user selection through the display unit 130 .

The display driving unit 131 provides a driving signal to the display unit 130 .

All operations of the image capturing apparatus 100 may be controlled by the controller 120 . The controller 120 provides control signals for the operation of each component to the lens driver 10 , the focus lens driver 9 , and the imaging sensor controller 7 .

The controller 120 may process an input image signal and control each component according to the input image signal or an external input signal. The controller 120 may correspond to one or a plurality of processors. The processor may be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory in which a program executable in the microprocessor is stored. In addition, it can be understood by those of ordinary skill in the art to which this embodiment pertains that it may be implemented in other types of hardware.

The control unit 120 executes a program stored in the program storage unit 170 or has a separate module to generate a control signal for controlling auto focusing, zoom change, focus change, automatic exposure compensation, etc. It is provided to the driving unit 9 and the imaging sensor control unit 7, and may collectively control operations of components included in the photographing apparatus 100, such as a shutter and a strobe.

In addition, the controller 120 may be connected to an external monitor to perform a predetermined image signal processing on the image signal input from the image signal processing unit 141 to be displayed on the external monitor, and transmit the processed image data to display the image on the external monitor. This can be displayed.

According to an embodiment, the controller 120 may cause the photographing device 100 to photograph an object using a virtual image by executing one or more instructions of a program stored in the program storage unit 170 . The program stored in the program storage unit 170 will be described later with reference to FIG. 3B.

The operation unit 180 is where the user can input a control signal. The control unit 180 includes a shutter-release button 181 that inputs a shutter-release signal to take a picture by exposing the image sensor 6 to light for a set period of time, a zoom button 182 that widens the angle of view or narrows the angle of view according to the input, and a power supply. It may include various function buttons, such as a power button 183 for inputting a control signal for on-off control, other mode selection buttons, and other shooting settings adjustment buttons. The manipulation unit 180 may be implemented in any form through which a user can input a control signal, such as a button, a keyboard, a touch pad, a touch screen, and a remote controller.

The release button 181 has a two-stage switch structure including a switch SW1 for starting the shooting preparation of the video image pickup device and a switch SW2 for starting shooting. The zoom button 182 is an operation switch for changing the focal length of the photographing apparatus. Then, based on the results of the phase difference AF and contrast AF, the focus lens 3 is controlled by the AF control unit 121 via the focus lens driving unit 9 . In addition, according to the operation of the zoom button, the control unit 120 controls the photographing lens 1 related to the change of the focal length through the lens driving unit 10 . In this case, since the position of the photographing lens 1 is transmitted from the focal length detector 11 to the controller 120, the focal length of the photographing lens 1 may always be detected.

The communication unit 190 performs pairing with an external device using at least one of Wi-Fi, Bluetooth, Bluetooth low energy, infrared communication, or laser beam communication, according to an embodiment, so as to use an external device such as a smart watch or smart glasses for a shooting guide. information can be transmitted. Also, according to an embodiment, the communication unit 190 may receive a virtual image by accessing a server providing content using at least one of 3G, 3GPP, 4G, and Wi-Fi.

FIG. 3B is a block diagram of a module that may be stored in the program storage unit 170 shown in FIG. 3A to perform a method of operating a photographing apparatus according to an exemplary embodiment.

Referring to FIG. 3B , the program storage unit 170 includes a virtual image determination module 171, an image acquisition module 172, a virtual image configuration module 173, a photographing module 174, a photographing accessory management module 175, a posture detection module 176, a determination module 177, focusing and photographing a module 178; an image combining module 179;

The virtual image determination module 171 is optional.

The virtual image determining module 1701 may include one or more instructions for determining a 3D virtual image used to represent a photographed object. The virtual image determining module 171 may include one or more instructions for selecting a 3D virtual image used to represent a photographing object according to a user input. The photographing apparatus may represent the photographing object by using a 3D virtual image configured as a default. When a photographing object is represented using a 3D virtual image configured as a default, the virtual image determining module 171 is not required.

The image acquisition module 172 may include one or more instructions for acquiring a background image having depth information.

The virtual image configuration module 173 may include one or more instructions for configuring a 3D virtual image in the background image.

In one example, the virtual image configuration module 173 configures a three-dimensional space position and/or a three-dimensional posture of the three-dimensional virtual image.

In one example, the virtual image configuration module 173 configures the 3D spatial position of the 3D virtual image in the background image according to a user action. The user is notified whether the area in which each of the contents is located is a suitable area for placing the three-dimensional virtual image.

In one example, the virtual image construction module 173 configures a three-dimensional virtual image in the background image based on the photographing appendage of the three-dimensional virtual image. The relationship between the three-dimensional virtual image and the imaging appendage of the three-dimensional virtual image is defined by a pre-configured imaging scenario template.

In one example, the virtual image construction module 173 configures the three-dimensional posture of the three-dimensional virtual image by configuring the three-dimensional space position of the joining point of the three-dimensional virtual image.

The imaging module 174 may include one or more instructions for performing imaging based on the configured 3D virtual image. That is, after the photographing composition is completed using the 3D virtual image, the photographing module 174 performs photographing based on the configured 3D virtual image.

Preferably, the imaging module 174 also guides the imaging object to perform adjustment based on the three-dimensional virtual image configured in the background image. The photographing module 174 includes a difference determining module 174-1 and a photographing guide module 174-2. The difference determining module 174-1 may include one or more instructions for determining a difference between a three-dimensional virtual image in a background image and a photographing object. can

In an example, the difference determining module 174-1 detects a photographing object based on a 3D posture of the 3D virtual image, detects a photographing object based on a 3D spatial position provided by the wearable device, or a user selected A photographing object is determined based on target tracking for the object.

In one example, the difference between the 3D virtual image and the photographing object in the background image includes a 3D spatial position difference and/or 3D posture difference between the 3D virtual image and the photographing object in the background image.

In one example, the difference determining module 174-1 is configured to determine the 3D posture difference between the imaging object and the 3D virtual image, the 3D space position of the bonding point of the imaging object and the 3D space of the corresponding bonding point of the 3D virtual image Determined by comparing positions.

The photographing guide module 174-2 may include one or more instructions for outputting a photographing guide based on the determined difference and performing photographing.

In one example, the photographing guide module 174-2 outputs the photographing guide through the photographing apparatus and/or the wearable device.

Preferably, the imaging module 174 focuses and captures a photographing object when appropriate. In one example, the capturing module 174 focuses and captures the capturing object when the capturing object matches the 3D virtual image in the background image.

In one example, the three-dimensional virtual image has a posture used to predict the appearance of a posture of a photographing object desired to be captured, and the imaging module 174 automatically focuses and captures the photographing object.

In one example, the photographing object includes a plurality of objects, and for each object, the photographing module 174 focuses and captures the object when the object matches the three-dimensional virtual image in the background image. The photographing apparatus further includes an image combining module (not shown) that extracts the objects from the photographed images corresponding to the objects, and combines all the extracted objects with a background image.

In one example, the three-dimensional virtual image has multiple postures used to represent a series of photographing object postures desired to be continuously photographed, and when the photographing object continuously matches the three-dimensional virtual image in the background image for a preset time period, The photographing module 174 automatically focuses on the photographing object and continuously captures the photographing object.

In one example, the shooting object includes a plurality of objects, and before the shooting object is focused and captured, the object that does not match the three-dimensional virtual image is removed from the shooting preview image; Alternatively, after a photographing object is focused and photographed, an object that does not match the 3D virtual image is removed from the photographed image.

In one example, the program storage unit 170 may further include a photographing accessory management module 175 . The photographing accessory management module 17 further includes a photographing accessory determining module 175-1 and a photographing accessory configuration module 175-2. The imaging accessory determining module 175 - 1 may include one or more instructions for determining an imaging accessory. The imaging accessory configuration module 175-2 may include one or more instructions for configuring the 3D spatial position and/or the 3D posture of the imaging accessory in the background image. The posture detection module 176 may include one or more instructions for detecting the posture of the photographed object in the background image.

The determination module 177 may include one or more instructions for determining whether the posture of the photographing object matches the focusing posture.

In one example, the focusing posture is a pre-configured posture used to indicate a desired shooting posture of the virtual image of the imaging object, or any shooting posture in the pre-configured imaging posture database, or is the best shooting posture of the imaging object.

The focusing and photographing module 178 may include one or more instructions for focusing and photographing the photographing object when the detected posture of the photographing object matches the focusing posture.

In one example, the focusing and capturing module 178 may include one or more instructions for automatically focusing and capturing a photographing object.

In one example, the focusing posture is a posture used to predict the appearance of a posture of a photographing object desired to be captured, and the focusing and photographing module 178 automatically focuses and captures the object.

In one example, the photographing object includes a plurality of objects, and the focusing and photographing module 178 focuses and captures the object when the posture of the photographing object matches the focusing posture with respect to each object.

The program storage unit 170 may further include an image combining module 179, and the image combining module 179 may include one or more instructions for extracting objects from corresponding captured images and combining all the extracted objects with a background image.

In one example, the focusing posture includes postures used to indicate a series of photographing object postures desired to be continuously photographed, the posture detection module 176 continuously detects the posture of the photographing object in the background image, and the focusing and photographing module 178 If it is detected that the posture of the photographing object continuously coincides with the focusing posture for a preset time period, the camera automatically focuses the photographing object and continuously shoots the photographing object.

In one example, the photographing object includes a plurality of objects, and the focusing and photographing module 178 removes an object that does not match the focusing posture from the photographing preview image before the photographing object is focused and photographed, or the photographing object is focused After being captured and taken, objects that do not match the focusing posture are removed from the captured image.

<Overall action>

4 is a flowchart illustrating a photographing method according to an exemplary embodiment.

Referring to FIG. 4 , in operation 410 , the photographing device 100 may determine a 3D virtual image used to represent a photographing object. According to an example, the 3D virtual image may be selected according to a user input. Optionally, the photographing device 100 may represent the photographing object using a 3D virtual image configured as a default. If a 3D virtual image configured as a default is used to represent a photographing object, the procedure of operation 410 may not be performed.

In operation 420, the photographing device 100 may acquire a background image having depth information.

The background image may be an actual image of a scenario in which shooting is performed, that is, a scenario image in which shooting is completed, or a preview image of a scenario in which shooting is performed, ie, a preview scenario image captured by the camera in real time.

In operation 430, the photographing device 100 may arrange a 3D virtual image on the background image. For example, the photographing device 100 configures a 3D virtual image on a background image having depth information. The shooting effect of the shooting object may be simulated so that the photographer and/or the shooting object achieve the shooting effect in advance before performing the shooting.

In operation 440, the photographing device 100 may photograph an object based on the configured 3D virtual image. That is, after composing a photographing composition is performed in advance using the 3D virtual image, the photographing may be performed based on the configured 3D virtual image.

<Determination of 3D virtual image>

A procedure for determining a 3D virtual image used to represent a photographed object in operation 410 will be described in detail below.

According to an embodiment, the 3D virtual image may be a 3D character model of a specified posture. For example, the 3D virtual image may be a 3D virtual image of a user, a cartoon, a celebrity, a celebrity, a movie cast, and the like.

5 illustrates a three-dimensional virtual image according to an embodiment. 510 of FIG. 5 illustrates a 3D virtual image of a cartoon, 520 of FIG. 5 illustrates a 3D virtual image of a celebrity, and 530 of FIG. 5 illustrates a 3D virtual image of a movie cast.

According to an embodiment, the 3D virtual image may be a pre-stored 3D virtual image or a 3D virtual image prepared according to another image. For example, a database for storing a predetermined amount of 3D virtual images may be built so that a user can select a 3D virtual image from among previously stored 3D virtual images. Optionally, the three-dimensional virtual image may be obtained through the Internet or the like according to a user's manipulation. Optionally, the 3D virtual image may be produced through 3D modeling according to the image selected by the user. The image selected by the user may be a locally stored image, an image obtained through the Internet, etc., an image photographed by the user as a target person, or an image photographed by another person as a target person. If the image includes a plurality of people, the user may select a target person to create a three-dimensional virtual image. For example, the user may select an image including person A, person B, and person C as an image stored in the local storage, then select person A as a target person to create a corresponding three-dimensional virtual image. The 3D virtual image obtained through the Internet, etc. and the 3D virtual image produced according to the image provided by the user can be stored in the shooting posture database, so that the user can perform the next shooting using the 3D virtual image .

Moreover, the photographed image of the photographing object taking a specific photographing posture may be automatically stored in the photographing posture database as a three-dimensional virtual image. For example, according to user approval, it is determined whether the photographed image of the photographing object is stored in the common photographing posture database. In the photographing posture database, the image of the photographing object may be displayed in such a way that features such as appearance are not displayed.

If the photographing object is a single object, a single three-dimensional virtual image or a three-dimensional virtual image with a series of postures may be determined. When there are a plurality of photographing objects, each of a plurality of 3D virtual images may be determined or a combination of a plurality of 3D virtual images may be determined.

According to an embodiment, in a continuous photographing state in which a series of postures of a photographing object are continuously photographed, a three-dimensional virtual image having a series of postures may be determined. For example, the photographing posture database stores a posture sequence of a three-dimensional virtual image, and the three-dimensional virtual image and a series of postures of the three-dimensional virtual image may be determined by selecting the posture sequence. Optionally, after the three-dimensional virtual image is first determined, a series of postures of the three-dimensional virtual image may be selected from several different postures of the three-dimensional virtual image. Moreover, when capturing the user (ie, the desired pose of the imaging object is captured), a three-dimensional virtual image with a series of poses can be determined. For example, a posture sequence of a three-dimensional virtual image may be first determined and then a posture desired to be captured may be determined from the posture sequence. Therefore, the predetermined posture is used to predict the occurrence of a desired posture to be photographed before the desired posture is obtained in the posture sequence, so that when it is determined that the posture of the photographing object matches the predetermined posture of the photographing object, the photographing object is focused, photographed, and captured An image of the imaging object in the desired posture may be captured.

According to an embodiment, when the photographing object includes a plurality of objects, a single 3D virtual image may be continuously determined for each object. Optionally, a three-dimensional virtual image with a series of poses is determined for each of the objects. Accordingly, a plurality of three-dimensional virtual images are respectively determined for a plurality of objects. In another example, after a combination comprising a plurality of three-dimensional virtual images is determined, it is determined which three-dimensional virtual image in the combination has been determined for each object. For example, the photographing posture database may store a plurality of combinations including a plurality of three-dimensional virtual images, and the postures of the plurality of three-dimensional virtual images in the combination may be related to each other (eg, of the plurality of three-dimensional virtual images). Postures can be organized with each other).

According to an embodiment, the photographing posture database may further include a photographing scenario template. The shooting scenario template includes a three-dimensional virtual image having a limited three-dimensional spatial position and a limited three-dimensional posture. When the shooting scenario template is selected, the 3D spatial position and 3D posture are not additionally configured, and imaging may be directly performed based on the 3D virtual image in the shooting scenario template. For example, after the shooting scenario template is selected, the 3D virtual image in the shooting scenario template may be changed, and the changed 3D virtual image is the 3D spatial position and 3D posture of the original 3D virtual image in the shooting scenario template can still have

In addition, the shooting scenario template may also include an auxiliary option (eg, a shooting-related object) as well as a three-dimensional virtual image. Therefore, the shooting scenario template can represent a corresponding shooting scenario by limiting several relationships between the three-dimensional virtual image and the auxiliary options. For example, some postures photographed with the help of a location may be supported by a specific environment or a specific object (eg, the sun, a statue, a building, etc.). Such an environment or object is a shooting-related object. The shooting scenario template may be an actual image, a schematic diagram indicating a combination of a person posture skeleton and an actual background, or a schematic diagram indicating a human posture skeleton and a photographing-related object. The traditional photographing posture may be obtained through the Internet or the like, and the photographing posture supported by the environment or object may be determined from the traditional photographing posture. This environment or object is marked as a photographing-related object, and properties (eg, color, sharpness) of the photographing-related object are stored in the photographing posture database.

According to an embodiment, the photographing device 100 may automatically determine the 3D virtual image used to represent the photographing object as a default in the photographing device, or may input the 3D virtual image used to represent the photographing object to the user input. So you can choose.

In an example of selecting a three-dimensional virtual image used to represent a photographed object according to a user input, the three-dimensional virtual image used to represent a photographed object is a user voice operation, a physical button operation, a touch screen operation, a gesture It may be selected according to at least one of an action, an action for an augmented-reality human-computer interaction interface, and an action for an external controller.

In an example of selecting a three-dimensional virtual image according to a user's voice operation, when a voice command to "select a first three-dimensional virtual image" is received, voice recognition is performed on the voice command to perform a first A three-dimensional virtual image of

In an example of selecting a 3D virtual image according to a physical button operation, the manipulated physical button may be a home button (a main screen button), a volume adjustment button, a power button, and other newly added buttons. For example, a user may select a 3D virtual image by moving a selection box through a volume adjustment button and then select it through a home button.

In the example of selecting a three-dimensional virtual image according to a touch screen operation, the user may click in a predefined manner a specific three-dimensional virtual image or a selection box used to select this particular three-dimensional virtual image. The predefined manner may be a short press, a long press, a short press for a preset number of times, and an alternating short press and a long press. Optionally, the user may drag a specific three-dimensional virtual image.

6 illustrates an example of selecting a 3D virtual image according to an exemplary embodiment.

Referring to FIG. 6 , a smartphone is illustrated as an example of the photographing device 100 . The display unit 130 of the photographing apparatus 100 may be divided into a display area 130a and a menu area 130b. The display area 130a may be used to display the acquired background image, the determined 3D virtual image, and the photographed object. The menu area 130b may be used to provide a menu according to a photographing method according to embodiments and to allow a user to select it. The menu area 130b may display a <select 3D virtual image> message and one or more 3D virtual images that the user can select. The user may select a combination 610 including two 3D virtual images in the menu area 130b. . Accordingly, the two 3D virtual images may be selected according to a user action of selecting a combination of the two 3D virtual images and dragging them to the display area 130a.

In an example of selecting a 3D virtual image according to a user gesture operation, the gesture operation may be performed with one hand or both hands. A gesture action is an action such as waving a hand, drawing a circle with the hand, drawing a square with the hand, drawing a triangle with the hand, beckoning up, beckoning down, beckoning left, or beckoning right. can The user gesture may be detected and recognized through the current posture detection device.

7 illustrates another example of selecting a 3D virtual image according to an exemplary embodiment.

Referring to FIG. 7 , 3D virtual images used to represent two photographed objects are continuously selected. When a three-dimensional virtual image corresponding to each photographing object is selected, a plurality of three-dimensional virtual images may be presented for the user's selection, and a selection box 710 may indicate an up gesture, a downward gesture, a left gesture, or a right gesture. is selected according to a user gesture action (e.g., when a user gesture of gesturing upwards is detected, the selection label moves toward the selection box of the three-dimensional virtual image to be selected), and the selection is made according to the user gesture action of drawing a circle with the hand. . Moreover, the selection may be made via voice action 720.

According to an embodiment, the 3D virtual image may be selected according to a user action for an external controller (eg, an action for a handwriting pen associated with a photographing apparatus, a device having a remote control function, such as a wearable device) action for). That is, the three-dimensional virtual image is selected according to a command from an external controller. For example, when the imaging device is connected to the handwriting pen, a 3D virtual image is selected according to a command from the handwriting pen. The user can select a 3D virtual image by moving a selection box for selecting the 3D virtual image using a button of the handwriting pen, and clicking the selected 3D virtual image with the handwriting pen.

In operation 410, regardless of the type of the determined 3D virtual image or the method in which the 3D virtual image is determined, in order to change the already selected 3D virtual image, the user can determine the 3D virtual image again in a subsequent step have to understand

<Acquisition of background image>

A procedure for obtaining a background image with depth information is described in operation 420 and is described in detail below.

The background image may be an actual shot image of a scenario in which shooting is performed (ie, a shot scenario image), or may be a preview image of a scenario in which shooting is performed (ie, a preview scenario image captured in real time by the camera). .

The background image with depth information may be obtained through a depth camera of a photographing apparatus, or may be obtained through a camera array including two or more cameras. Two or more cameras in the camera array have overlapping view angles to obtain depth information.

The procedure of operation 410 and the procedure of operation 420 may be performed sequentially, or the procedure may be performed first in operation 420 and then the procedure may be performed after that in operation 410, or the procedure in operation 410 and operation 420 may be performed simultaneously, It should be understood that the order of performing the procedures is not limited in the present invention.

In an embodiment, a three-dimensional virtual image used to represent a photographing object may be first determined, and then, a background image with depth information may be obtained. For example, after the camera application is executed, an interface used to select a three-dimensional virtual image is presented to the user, and thus the user selects a three-dimensional virtual image used to represent a photographing object. After the user selects the 3D virtual image, a background image having depth information is obtained.

In another embodiment, a background image with depth information may be determined first, and then a three-dimensional virtual image used to represent a photographing object may be determined. For example, after the camera application is executed, a background image with depth information is obtained, and then an interface used to select a three-dimensional virtual image is presented to the user, so that the user can use 3 You can choose a dimensional virtual image.

<Composition of 3D virtual image to background image>

Hereinafter, the procedure of operation 430 of configuring the 3D virtual image in the background image will be described in detail.

In one embodiment, the shooting effect of the shooting object can be simulated by composing a three-dimensional virtual image on a background image with depth information, so that the photographer and/or the shooting object can know the shooting effect in advance before shooting is performed .

In particular, after the three-dimensional virtual image used to represent the photographing object is determined, the determined three-dimensional virtual image is in a movable state. This three-dimensional virtual image may be configured in the background image according to user input, or may be automatically configured in the background image. In one example, a three-dimensional spatial position and/or a three-dimensional posture of the three-dimensional virtual image may be configured.

When the 3D spatial position of the 3D virtual image is configured in the background image, each of the positions of the 3D virtual image may be configured along a horizontal direction, a vertical direction, and a depth direction. The three-dimensional spatial position of the three-dimensional virtual image may be roughly configured on the background image, or precisely configured on the background image, or finely adjusted after being roughly configured on the background image, but the present invention is not limited thereto.

In one embodiment, the three-dimensional spatial position of the three-dimensional virtual image may be approximately constructed automatically in the background image, or may be approximately constructed in the background image according to a user action. If the three-dimensional spatial position of the three-dimensional virtual image is roughly configured in the background image according to a user action, the selected three-dimensional virtual image may be moved to a corresponding position in the background image according to the user action (eg, dragging, etc.). At this point, the posture of the 3D virtual image is the default posture, and subsequently, the user can precisely adjust the posture of the 3D virtual image. As shown in FIG. 6 , a 3D virtual image is constructed on the position of the background image through a user operation of dragging the selected 3D virtual image to the position of the background image. Optionally, the three-dimensional virtual image is constructed on a user-specified location within the background image. For example, when the selected three-dimensional virtual image is in a situation to be positioned, by a user action of clicking on the position of the background image, the three-dimensional virtual image may be configured at a specific position in the background image. Moreover, the three-dimensional spatial position of the three-dimensional virtual image can be roughly constructed in the background image through different user actions.

In one embodiment, the user may be queried whether the region in which each piece of content is located is suitable for placing a three-dimensional virtual image. For example, before the user configures the three-dimensional spatial position of the three-dimensional virtual image in the background image, the user may be queried whether the region in which each part of the content is located is suitable for placing the three-dimensional virtual image. Therefore, when the user configures the 3D spatial position of the 3D virtual image on the background image, the user can know an appropriate position and an inappropriate position for arranging the 3D virtual image. In another embodiment, when the user places the three-dimensional virtual image in an unsuitable location for placing the three-dimensional virtual image, the user is notified that the area is not suitable for placing the three-dimensional virtual image, or The user may be notified of how to place the three-dimensional virtual image in an area suitable for placing the three-dimensional virtual image.

8 and 9 show examples of notifying a user of a location area according to an embodiment.

As shown in FIG. 8 , a region (eg, the ground) 810 suitable for placing a 3D virtual image may be marked. Optionally, a guiding message "Place the 3D virtual image in an area suitable for placing the 3D virtual image" to guide the user to place the 3D virtual image in an area suitable for placing the 3D virtual image 820 may be notified to the user.

As shown in FIG. 9 , regions 910 and 930 that are not suitable for placing a three-dimensional virtual image (eg, a region with water or a region with an obstruction) may be marked. Optionally, when the user places the 3D virtual image in an area where the 3D virtual image is not suitable for placing the 3D virtual image, "The area is not suitable for placing the 3D virtual image. The 3D virtual image is not suitable for placing the 3D virtual image. Place a three-dimensional virtual image on the area." a guidance message 920 may be notified to the user.

In one embodiment, each piece of content in the background image may be analyzed to inform the user whether the area in which each piece of content is placed is a suitable area for placing a three-dimensional virtual image. For example, according to a pre-established scenario element database, in order to determine whether a region in which the content part is located is suitable for placing a three-dimensional virtual image, the content in the background image is divided into a plurality of parts segmented, and the properties of each part of the content are analyzed to determine whether the content is water, obstructions, land, etc. In addition, it is possible to manually mark an area that is not suitable for placing a three-dimensional virtual image on the background image.

In an example of automatically roughly configuring the three-dimensional spatial position of the three-dimensional virtual image in the background image according to an embodiment, the three-dimensional virtual image may be automatically constructed on an appropriate position in the background image. For example, a suitable location may be a central location of the background image, or a location where the entire three-dimensional virtual image can be placed within the background image. In this case, the posture of the 3D virtual image may be a default posture, and subsequently, the user may precisely adjust the posture of the 3D virtual image.

According to an embodiment, the 3D spatial position and/or 3D posture of the 3D virtual image may be automatically configured in the background image according to the shooting scenario template. For example, if the selected shooting scenario template does not include a shooting-related object, a three-dimensional virtual image is automatically constructed in an appropriate position, and a three-dimensional posture of the three-dimensional virtual image is automatically constructed according to the posture of a person in the template; After that, the user can precisely adjust the three-dimensional spatial position and/or the three-dimensional posture of the three-dimensional virtual image. In addition, since the shooting scenario limits the 3D spatial position and 3D posture of the person, the 3D spatial position and 3D posture of the 3D virtual image are automatically determined according to the 3D spatial position and 3D posture of the person in the shooting scenario template. can be composed of

In another embodiment, if the selected shooting scenario template includes a shooting-related object, the three-dimensional posture may be configured based on the shooting-related object of the three-dimensional virtual image in the background image, and between the shooting-related object and the 3D virtual image Relationships may be limited to pre-configured shooting scenario templates. In this situation, the user is asked to select a shooting-related object in the background image, or the shooting-related object in the background image may be automatically detected according to the corresponding shooting-related object in the shooting scenario template. For example, the properties of the shooting-related object in the shooting scenario template are studied according to a machine learning method, or various parts of the content in the background image and shooting-related objects in the shooting scenario template are In order to determine whether a photographing-related object that matches the object is included, they may be compared with each other. If it is determined that the background image does not include a shooting-related object that matches the shooting-related object in the shooting scenario template, the background image is re-acquired, or the user may be required to manually select the shooting-related object. If it is determined that the background image includes a shooting-related object that matches the shooting-related object in the shooting scenario template, an appropriate 3D spatial position and/or 3D posture of the 3D virtual image is automatically set in the background image according to the shooting-related object is composed

For example, the three-dimensional spatial position and/or three-dimensional posture of the three-dimensional virtual image in the background image is dependent on the size, three-dimensional spatial position and/or three-dimensional posture of the imaging-related object in the three-dimensional virtual image within the background image. Based on the configuration, the 3D spatial positional relationship and/or the 3D posture relationship between the 3D virtual image and the shooting-related object of the 3D virtual image are limited to a preset shooting scenario template. For example, the size of the 3D virtual image in the background image may be determined according to a size ratio of the shooting-related object in each of the shooting scenario template and the background image. The three-dimensional spatial position and/or three-dimensional posture of the three-dimensional virtual image in the background image is the three-dimensional spatial positional relationship between the three-dimensional virtual image and the photographing-related object in the photographing scenario template, and the three-dimensional spatial position of the photographing-related object in the background image , and the size of the 3D virtual image in the background image.

10 illustrates an example of automatically configuring a 3D spatial position of a 3D virtual image according to an exemplary embodiment. As shown in FIG. 10 , a <Catch the sun by hand> shooting scenario template is selected according to a user action, and the user is requested to select a <sun>, which is a shooting-related object. After <Sun> is selected, at least one item among the three-dimensional space position, three-dimensional posture, and size of the three-dimensional virtual image is automatically calculated, and the three-dimensional space position and/or three-dimensional posture of the three-dimensional virtual image is constructed within the background image. If the position of the <sun> in the background image is not appropriate, the user is notified of the configuration failure, and the reason is displayed. In addition, after the position of the three-dimensional virtual image is automatically configured, the configured position may be further adjusted according to a user's manual operation.

In the example of precisely configuring the 3D spatial position of the 3D virtual image in the background image, the 3D spatial position of the 3D virtual image in the background image may be precisely configured according to various user actions. For example, a precise configuration may be implemented according to a user action, such as a voice action, a physical button action, a touch screen action, a gesture action, an augmented reality user-computer interaction interface, an action on an external controller, and the like.

According to an embodiment, the 3D spatial location of the 3D virtual image in the background image may be configured according to a user's voice motion.

In the example of configuring the 3D spatial position of the 3D virtual image in the background image according to the user's voice motion, the user's voice to move the 3D virtual image 1 meter to the left and 3 meters to the rear with respect to the camera When the command is received, voice recognition is performed on the voice command, and it is decided to move the three-dimensional virtual image from the background image to the left by 1 meter and to the rear by 3 meters. In addition, when a simple user command of <Move 4m to the right> is received, voice recognition is performed for the user command, and the user's voice command is used to move the 3D virtual image to the right 4 meters from the background image. is decided

According to an embodiment, the 3D spatial position of the 3D virtual image in the background image may be configured according to a user's physical button operation.

In the example of configuring the 3D spatial position of the 3D virtual image in the background image according to the user's physical button operation, the operating physical button is a home button (main screen button), a volume control button, a power button, and other newly added buttons. can be For example, the user may determine a moving direction (ie, one of a horizontal direction, a vertical direction, and a depth direction) by simultaneously pressing a volume up button and a volume down button in the volume adjustment button. For example, the initial default movement direction is the horizontal direction. The user can select the vertical direction as the movement direction by simultaneously pressing the volume up button and the volume down button in the volume control button, and then press the volume up button and the volume down button again at the same time to select the depth direction as the movement direction. The currently selected moving direction may be notified to the user in a manner such as text, icon, or the like. After determining the moving direction, the user may independently press the volume up button or the volume down button to move the position of the 3D virtual image in the determined moving direction. For example, individually pressing the volume up button moves the 3D virtual image to the left, pressing the volume down button individually to move the 3D virtual image to the right, pressing the volume up button individually to move the 3D virtual image upward press the volume down button individually to move the 3D virtual image down, press the volume up button individually to move the 3D virtual image forward, and individually press the volume down button to move the 3D virtual image backwards move After the configuration is complete, press the Home button to make the decision.

According to an embodiment, the 3D spatial position of the 3D virtual image in the background image may be configured according to a user's touch screen operation.

In the example of configuring the 3D spatial position of the 3D virtual image in the background image according to the user's touch screen operation, the user moves the horizontal position and the vertical position of the 3D virtual image by dragging the determined 3D virtual image with a single finger. Alternatively, the horizontal and vertical positions of the 3D virtual image can be configured by sliding horizontally and vertically on the screen, or the depth of the 3D virtual image through touch actions such as two-finger pinch and two-finger spread. The location is configurable. Similarly, other directions can be adjusted with a touch action such as short press, long press, short press a preset number of times, alternating short press and long press. In addition, the user can configure the 3D spatial position of the 3D virtual image in the background image by sliding the scroll bar on the user interface or inputting the corresponding content into the text box.

11 illustrates an example of configuring a 3D spatial position of a 3D virtual image according to an exemplary embodiment .

Referring to FIG. 11 , the user selects the 3D spatial position of the selected 3D virtual image in the background image, for example, by dragging the 3D virtual image with a single finger 1110 , pinching and/or spreading two fingers 1120 , scroll bar It can be configured through sliding 1130, inputting content corresponding to the text box 1140, and voice operation 1150. Also, a notification 1160 about the current 3D spatial location of the selected 3D virtual image may be output to the user.

If the 3D spatial position of the 3D virtual image in the background image is configured according to a user gesture operation, the gesture operation may be performed with one hand or both hands. The gesture action may be an action such as waving one hand, drawing a circle by hand, drawing a rectangle by hand, or drawing a triangle by hand. For example, the user may move the selected three-dimensional virtual image in a corresponding direction through gesture actions such as hand shaking, hand shaking, hand shaking left, hand shaking right, hand shaking rightward, and hand shaking left down. can be moved After that, the end of the movement is determined through a gesture operation of drawing a circle with the hand. A user gesture action may be detected and recognized through an existing gesture detection device.

The three-dimensional spatial position of the three-dimensional virtual image in the background image is dependent on a user action for an external controller (eg, an action for a handwriting pen associated with an imaging device, an action for a device having a remote control function, such as a wearable device). Thus, it can be configured. For example, when the photographing device is connected to a handwritten pen, a 3D virtual image is selected according to a command from the handwritten pen, and then, the user uses a button of the handwritten pen and slides the point of the handwritten pen to A 3D virtual image can be moved. For example, in order to drag the 3D virtual image to a desired position, the user may press the 3D virtual image using the tip of the handwriting pen and slide the tip while simultaneously pressing the button of the handwriting pen, and the sliding tip is a photographing device displayed on the screen of

In addition, in a situation where the background image is a panoramic image, the panoramic image and the three-dimensional virtual image are displayed to the user from various angles (for example, the panoramic image and the three-dimensional virtual image can be rotated to the left by 90 degrees at the same time), and the user can The three-dimensional spatial position of the three-dimensional virtual image can be conveniently configured on the background image. For example, the user may obtain the position in the depth direction of the 3D virtual image according to a profile image obtained by simultaneously rotating the panoramic image and the 3D virtual image to the left by 90 degrees.

In the example of configuring the three-dimensional posture of the three-dimensional virtual image in the background image according to the user's motion, the three-dimensional posture of the three-dimensional virtual image is the rotation angle of the three-dimensional virtual image and/or the three-dimensional space of the three-dimensional virtual image joining point. It can be configured by configuring the location. The rotation angle of the 3D virtual image may be configured along the rotation axis in three directions, respectively. For example, the midpoint, that is, the center of the 3D virtual image is configured as an origin of rotation, and the rotation may be performed in a corresponding plane. Optionally, the origin of rotation is user configurable.

The rotation angle of the 3D virtual image may be configured according to various user operation methods. For example, the rotation angle is configured according to a user action, such as a user voice action, a physical button action, a touch screen action, a gesture action, an action for an augmented-reality human-computer interaction interface, an action for an external controller.

According to an embodiment, the rotation angle of the 3D virtual image may be configured according to a user's voice motion.

In the example of configuring the rotation angle of the 3D virtual image according to the user's voice motion, when a voice command to <rotate the center of the 3D virtual image to the origin by 10 degrees clockwise> is received from the user, the voice command is Speech recognition is performed on the object, and the center of the three-dimensional virtual image is rotated 10 degrees clockwise in the corresponding plane as the origin.

According to an embodiment, the rotation angle of the 3D virtual image may be configured according to a user's physical button operation.

In an example in which the rotation angle of the 3D virtual image is configured according to a user's physical button operation, the operated physical button may be a home button, a volume adjustment button, a power button, or other newly added buttons. For example, the user may determine the rotation axis (ie, the central horizontal axis, the central vertical axis, and the central depth axis) by simultaneously pressing the volume up button and the volume down button in the volume adjustment button. In this case, the currently selected rotation axis may be notified to the user in a manner such as text or icon. After the rotation axis is determined, the user can adjust the rotation angle by individually pressing the volume up button or the volume down button. For example, the user performs rotation along the central central vertical axis by pressing the volume up button and the volume down button in the volume control button at the same time, and rotates the 3D virtual image clockwise by pressing the volume up button, or by pressing the volume down button You decide to rotate the 3D virtual image counterclockwise, and finally press the home button to make these decisions.

According to an embodiment, the rotation angle of the 3D virtual image may be configured according to the operation of the touch screen.

In an example in which the rotation angle of the 3D virtual image is configured according to a user's touch screen operation, the user may adjust each rotation angle by sliding a scroll bar or inputting content corresponding to a text box. In addition, the user can adjust each rotation angle by dragging the 3D virtual image.

12 illustrates an example of configuring a rotation angle of a 3D virtual image according to an embodiment.

Referring to FIG. 12 , a user may configure a rotation angle of a 3D virtual image in a background image through sliding a scroll bar 1210, inputting content corresponding to a text box 1220, voice operation 1230, and the like. Also, a notification 1240 indicating the current rotation angle of the selected 3D virtual image may be output to the user.

In an example in which the rotation angle of the 3D virtual image is configured according to a user gesture operation, the gesture operation may be performed with one hand or both hands. The gesture action may be an action such as waving a hand, drawing a circle with the hand, drawing a rectangle with the hand, or drawing a triangle with the hand. For example, the user may rotate the 3D virtual image through gesture actions such as up, down, left or right, up left, and down left. After that, it is determined whether the rotation is completed through a gesture motion of drawing a circle with the hand. A user gesture action may be detected and recognized through an existing gesture detection device.

Configuring the rotation angle of the three-dimensional virtual image according to a user action for an external controller (for example, an action for a handwriting pen associated with a photographing apparatus, an action for a device having a remote control function, such as a wearable device) In an example, when the imaging device is connected with the writing pen, a three-dimensional virtual image is selected according to a command from the writing pen, and thereafter, the user uses a button of the writing pen and sliding the tip of the writing pen, thereby making the three-dimensional virtual image can be rotated.

In addition, the user can configure the 3D spatial position and rotation angle of the 3D virtual image at the same time. For example, a user may simultaneously configure a 3D spatial position and a rotation angle according to various user actions. For example, the configuration is performed according to a user action, such as a voice action, a physical button action, a touch screen action, a gesture action, an action for an augmented-reality human-computer interaction interface, and an action for an external controller.

According to an embodiment, the 3D spatial position and the rotation angle of the 3D virtual image may be simultaneously configured according to a user's voice motion.

In the example of simultaneously configuring the 3D spatial position and rotation angle of the 3D virtual image according to the user's voice motion, <Move the 3D virtual image 1 meter to the left and 3 meters to the rear, and When a voice command to rotate > 10 degrees is received from the user, voice recognition is performed for the voice command, and the 3D virtual image is moved 1 meter to the left and 3 meters to the rear, and the 3D virtual image is returned to the origin. It is rotated 10 degrees along the clockwise direction.

According to an embodiment, the 3D spatial position and rotation angle of the 3D virtual image may be simultaneously configured according to a user's physical button operation.

In an example in which the three-dimensional spatial position and rotation angle of the three-dimensional virtual image are simultaneously configured according to a user's physical button operation, the operated physical button may be a home button, a volume adjustment button, a power button, or other newly added buttons. For example, the user may determine a rotation axis or a movement direction by simultaneously pressing a volume up button and a volume down button in the volume control button. In this case, the currently selected rotation axis or movement direction may be notified to the user in a manner such as text or icon. After the rotation axis or movement direction is determined, the user may individually press the volume increase button or the volume decrease button to adjust the movement of the 3D virtual image in the determined direction or the rotation of the 3D virtual image along the determined rotation axis.

According to an embodiment, the 3D spatial position and the rotation angle of the 3D virtual image may be simultaneously configured according to a user's touch screen operation.

In an example of simultaneously configuring the 3D spatial position and rotation angle of the 3D virtual image according to the user's touch screen operation, the user may perform the configuration by sliding a scroll bar or inputting content corresponding to a text box. In addition, the user may perform the configuration by dragging to move or rotate the 3D virtual image. For example, the user may move the 3D virtual image by dragging the 3D virtual image with one finger, or rotate the 3D virtual image by dragging the 3D virtual image with two fingers.

According to an embodiment, the 3D spatial position and the rotation angle of the 3D virtual image may be simultaneously configured according to a user gesture action.

In an example in which the three-dimensional spatial position and rotation angle of the three-dimensional virtual image are configured according to a user gesture operation, the gesture operation may be performed with one hand or both hands. The gesture action may be an action such as waving a hand, drawing a circle with the hand, drawing a rectangle with the hand, or drawing a triangle with the hand. For example, the user initiates the construction of the rotation angle of the three-dimensional virtual image through a gesture action of drawing a triangle with a hand, beckoning up, beckoning down, beckoning left or beckoning right, beckon to the left, beck to the bottom left. Rotate a three-dimensional virtual image through gestures such as hand gestures. The user starts constructing the three-dimensional spatial position of the three-dimensional virtual image through the gesture action of drawing a rectangle by hand, and then gestures upward, gestures downward, gestures left or gestures right, gestures upward, gestures downward, and The 3D virtual image is moved through the same gesture action. After that, it is determined that the construction is finished through the gesture action of drawing a circle with the hand.

According to an exemplary embodiment, the 3D spatial position and the rotation angle of the 3D virtual image may be simultaneously configured according to a user action for an external controller.

Configure the three-dimensional spatial position and rotation angle of the three-dimensional virtual image according to a user action for an external controller (for example, an action for a handwriting pen associated with an imaging device, an action for a device having a remote control function such as a wearable device) In this example, when the imaging device is connected with the writing pen, a 3D virtual image is selected according to a command from the writing pen, and then the user uses the button of the writing pen and sliding the tip of the writing pen to create a 3D virtual image. The image may be moved and/or rotated.

When the three-dimensional posture of the three-dimensional virtual image is configured by configuring the three-dimensional spatial position of the joint point of the three-dimensional virtual image, the joint point of the three-dimensional virtual image may be selected according to various user actions. For example, a binding point may be selected according to a user action such as a voice action, a physical button action, a touch screen action, a gesture action, an action for an augmented-reality human-computer interaction interface, or an action for an external controller.

According to an embodiment, a joint point of a 3D virtual image may be selected according to a user's voice operation.

In an example of selecting a joint point of a 3D virtual image according to a user's voice operation, when a voice command to <select the left hand> is received from the user, voice recognition is performed for the voice command, and the left hand in the 3D virtual image is A bonding point is selected.

According to an embodiment, a joint point of a 3D virtual image may be selected according to a user's physical button operation.

In an example of selecting the binding point of the 3D virtual image according to the user's physical button operation, the operated physical button may be a home button, a volume control button, a power button, or other newly added buttons. For example, the user moves a selection box through a volume adjustment button to select a joining point of a three-dimensional virtual image, and then presses the home button to determine the selection.

According to an embodiment, a combination point of a 3D virtual image may be selected according to a user's touch screen operation.

In the example of selecting the joining point of the 3D virtual image according to the user's touch screen operation, the user may click the joining point of the 3D virtual image in a predetermined manner to select the joining point of the 3D virtual image . The predetermined method may be a short press, a long press, a short press for a preset number of times, or alternately a short press and a long press.

13 illustrates an example of selecting a joint point of a 3D virtual image according to an exemplary embodiment.

Referring to FIG. 13 , in order to reduce the difficulty of selecting the joint point of the 3D virtual image, the joint point may be displayed in a manner that can be easily selected by the user. In addition, 3D virtual images 1300 taken at various angles are displayed, and the images are colored. Therefore, the user can easily select the joining point of the 3D virtual image. The joint point of the 3D virtual image may be selected through a user operation such as a clicking operation 1310 of the joint point of the 3D virtual image, an operation 1320 of selecting a selection box, and a user voice operation 1330 .

When the joint point of the 3D virtual image is selected according to a user gesture operation, the gesture operation may be performed with one hand or both hands. Gesture actions include waving hand, circle with hand, square hand with hand, triangle with hand, beckon up, hand beckon down, wave left hand, or shake hand with right hand. It may be an action such as lifting. For example, the user selects a selection box used to select a joint point of a three-dimensional virtual image through a user gesture action of up, down, left, or right, and then, by hand The selection is decided through the gesture action of drawing. A user gesture action may be detected and recognized through an existing gesture detection device.

In an example of selecting the binding point of the three-dimensional virtual image according to a user action for an external controller (eg, an action for a handwriting pen associated with a photographing apparatus, an action for a device having a remote control function, such as a wearable device), shooting When the device is connected with the handwriting pen, the joining point of the 3D virtual image is selected according to the command from the handwriting pen, and the user selects the 3D virtual image by moving the selection box using the button of the handwriting pen, and selects the 3D virtual image Click a selected joint point on the image to determine the selection.

After the joining point of the three-dimensional virtual image is selected, the selected joining point of the three-dimensional virtual image is in a movable state. The three-dimensional spatial position of the user-selected joint point of the three-dimensional virtual image is configured according to various user actions. That is, the selected bonding point is composed of positions along the horizontal direction, the vertical direction, and the depth direction in the background image, respectively. For example, the configuration may be performed through a user voice operation, a physical button operation, a touch screen operation, a gesture operation, an operation for an augmented-reality human-computer interaction interface, an operation for an external controller, and the like.

According to an exemplary embodiment, a 3D spatial location of a joining point of a 3D virtual image may be configured according to a user's voice motion.

In the example of configuring the three-dimensional spatial location of the joint point of the three-dimensional virtual image according to the user's voice motion, when a voice command to <raise the left hand 4cm up, and then move it back 3cm> is received from the user, the voice command is Speech recognition is performed for the three-dimensional virtual image, and the position of the joint point of the three-dimensional virtual image is moved up by 4 cm, and then the position of the joint point is moved back by 3 cm.

According to an embodiment, the three-dimensional space position of the joint point of the three-dimensional virtual image may be configured according to a user's physical button operation.

In an example in which the three-dimensional spatial position of the joint point of the three-dimensional virtual image is simultaneously configured according to the user's physical button operation, the operated physical button may be a home button, a volume adjustment button, a power button, or other newly added buttons. For example, the user may determine a moving direction (ie, one of a horizontal direction, a vertical direction, and a depth direction) by simultaneously pressing a volume up button and a volume down button in the volume adjustment button. In this case, the currently selected moving direction may be notified to the user in a manner such as text or icon. After the moving direction is determined, the user can individually press the volume up button or the volume down button to move the position of the selected bonding point in the determined direction. For example, the selected bonding point can be moved to the left by individually pressing the volume up button, and can be moved to the right by individually pressing the volume down button. After the configuration is complete, press the Home button to make the decision.

According to an embodiment, the three-dimensional space position of the joint point of the three-dimensional virtual image may be configured according to a user's touch screen operation.

In the example of configuring the three-dimensional spatial position of the joining point of the three-dimensional virtual image according to the user's touch screen operation, the user drags the selected three-dimensional virtual image with one finger to move the horizontal and vertical positions of the three-dimensional virtual image or by sliding along the horizontal direction and sliding along the vertical direction on the screen, a horizontal position and a vertical position of the 3D virtual image may be configured. The different directions of the character model bonding point are adjusted through touch actions such as short press, long press, short press for a preset number of times, alternate short press and long press, and the like. In addition, the user may configure the 3D spatial position of the selected bonding point in the background image by sliding a scroll bar or inputting content corresponding to a text box.

14 illustrates an example of configuring a 3D spatial position of a joining point of a 3D virtual image according to an embodiment.

Referring to FIG. 14 , the user selects the three-dimensional spatial location of the selected joint point of the three-dimensional virtual image, dragging the joint point with one finger 1410, sticking and/or spreading both fingers 1420, sliding the scroll bar 1430, text Through input of content corresponding to the box 1440 and voice operation 1330, the background image may be configured.

According to an exemplary embodiment, a 3D spatial location of a joining point of a 3D virtual image may be configured according to a user gesture operation.

In an example in which the three-dimensional spatial position of the joining point of the three-dimensional virtual image is configured according to a user gesture operation, the gesture operation may be performed with one hand or both hands. The gesture action may be an action such as waving a hand, drawing a circle with the hand, drawing a rectangle with the hand, or drawing a triangle with the hand. For example, the user may move the selected bonding point in a corresponding direction through a gesture operation such as a gesture upward, a downward gesture, a left gesture, or a right gesture. After that, the end of the movement is determined through a gesture operation of drawing a circle with the hand. A user gesture action may be detected and recognized through an existing gesture detection device.

According to an exemplary embodiment, a 3D spatial position of a joining point of a 3D virtual image may be configured according to a user action for an external controller.

The three-dimensional spatial position of the three-dimensional virtual image binding point is configured according to a user action for an external controller (eg, an action for a handwriting pen associated with an imaging device, an action for a device having a remote control function such as a wearable device) In this example, when the photographing device is connected with the writing pen, a binding point of the 3D virtual image is selected according to a command from the writing pen, and then, the user selects the selected binding point by using the button of the writing pen and It can be moved by sliding the tip of

In order to make the three-dimensional spatial location of the joining point configured by the user accurate and valid, the three-dimensional spatial location of the joining point is configured according to the user's motion, and the properties of the configured joining point may be considered. For example, if a joint point configured according to a user's motion is a mother joint point, a child joint point corresponding to the mother joint point may be moved along with the movement of the mother joint point. In addition, the feasibility of a user action used to construct the three-dimensional spatial position of the joint point is determined according to the attribute of the joint point, and the user is notified. For example, the feasibility of a user action is determined according to the property that the three-dimensional spatial position of the joining point is limited by the length of the corresponding skeleton.

15 illustrates an example of informing a user that a three-dimensional spatial location of a joining point is incorrectly configured, according to an embodiment. 15 , the three-dimensional spatial position 1510 of the joining point configured by the user does not meet the length limit of the corresponding skeleton, so the imaging device 100 sends a notification 1510 that the three-dimensional spatial position of the joining point is configured incorrectly. can be printed to the user.

16 illustrates an example of selecting and configuring a 3D virtual image and a joint point of the 3D virtual image together, according to an embodiment. Referring to FIG. 16 , the imaging device 100 includes a selection box 1610 for selecting a first 3D virtual image and a second 3D virtual image, and a selection box 1620 for selecting a joint point of each selected 3D virtual image. can be displayed. As shown in FIG. 16 , the second three-dimensional virtual image and the right-hand joint point of the second three-dimensional virtual image can be selected and configured together through a user interface, so that the three-dimensional virtual image and its three-dimensional virtual image The joining points of the images can be selected and configured together.

If the background image is a preview image of the scenario in which the shooting is performed (i.e. the camera captures the preview image of the scenario in real time), the preview image is captured in real time, so the preview image is sensitive to changes in the action of the user's hand. It changes accordingly (eg, the preview image shakes as the user's hand shakes lightly). In an embodiment, the 3D spatial position and/or 3D posture of the 3D virtual image constructed in the preview image may be changed according to the change of the preview image.

In addition, in the process of the user constructing the 3D virtual image, the current 3D spatial position and/or the 3D posture of the 3D virtual image may be fed back to the user in real time on the user interface, and the preview image in the user interface and Since color can be applied to the three-dimensional virtual image, the user can acquire a change that occurs according to a user's motion in real time.

According to the above method, the shooting effect of the shooting object can be effectively simulated by constructing a three-dimensional virtual image in the background image, and thus a complete composition can be implemented before shooting.

<Perform shooting based on the configured 3D virtual image>

Hereinafter, the procedure of operation 440 of performing imaging based on the configured 3D virtual image will be described in detail.

That is, after composing a photographing composition in advance using the 3D virtual image, the photographing is performed based on the configured 3D virtual image.

Preferably, in the photographing process based on the constructed three-dimensional virtual image, the photographing object is guided to perform adjustment based on the three-dimensional virtual image constructed on the background image.

A procedure of operation 440 of performing imaging based on the configured 3D virtual image is described in FIG. 17 .

17 is a flowchart illustrating a method of performing imaging based on a 3D virtual image constructed according to an exemplary embodiment.

17 , in operation 1710, the photographing device 100 may determine a difference between the 3D virtual image in the background image and the photographing object.

The photographing apparatus 100 may automatically detect a photographing object. In an embodiment, the imaging object is detected based on a three-dimensional posture of the three-dimensional virtual image in the background image. For example, the photographing object may be detected by detecting an object having a three-dimensional posture similar to or coincident with the three-dimensional virtual image in the background image. For example, in operation 430, if a 3D spatial position of the 3D virtual image in the background image is configured, the configured position may be detected in real time. When an object having a three-dimensional posture similar to or coincident with the three-dimensional virtual image in the background image is detected, the object is determined as a photographing object.

In another embodiment, the photographing object may be determined based on 3D spatial location information provided by the wearable device. For example, a photographing object at a corresponding position in the background image is detected according to the 3D spatial location information, and the 3D spatial location information is provided by a wearable device worn by the photographing object and connected to the photographing apparatus, or the photographing object An electronic device carried by the user and connected to the photographing apparatus may provide it.

In an embodiment, the photographing object may be determined by performing target tracking on the object selected by the user. For example, the user may select an object as a photographing object in the background image, and thereafter, target tracking may be performed on the selected photographing object to determine the photographing object.

According to an embodiment, the photographing object may be determined by another method. In an example, the photographing object may be detected through facial recognition. The image of the photographing object may be photographed in advance or obtained from a photo album. A photographing object may be detected based on an image of the photographing object.

In another example, an action trajectory of the photographing object may be tracked, and the photographing object may be detected according to the action trajectory. For example, the action trajectory of the object is tracked, and it is determined whether the action direction of the object is toward the position of the constructed three-dimensional virtual image. If the action direction points to the position of the constructed 3D virtual image, it is determined that the object is a shooting object. If the action direction is not toward the position of the constructed three-dimensional virtual image, it is determined that the object is not a shooting object.

When the photographing object includes a plurality of people, each photographing object may be individually detected according to the above-described method.

In an embodiment, the difference between the 3D virtual image and the shooting object in the background image is the difference in 3D spatial position between the 3D virtual image and the shooting object in the background image and/or between the 3D virtual image and the shooting object in the background image may include a difference in the three-dimensional posture of

In one example, the 3D spatial position difference between the 3D virtual image and the photographing object may be determined by comparing the 3D spatial position of the photographing object with the 3D spatial position of the 3D virtual image. For example, the 3D spatial position difference between the 3D virtual image and the shooting object is the 3D spatial position in the horizontal direction, vertical direction, and depth direction of the shooting object in the horizontal direction, vertical direction, and depth direction of the 3D virtual image. It can be determined by sequentially comparing the three-dimensional spatial position in .

In one example, the 3D posture difference between the 3D virtual image and the photographing object may be determined by comparing the 3D spatial position of the bonding point of the photographing object with the 3D spatial position of the corresponding bonding point of the 3D virtual image. Each of the three-dimensional spatial positions of the joining points of the imaging object is sequentially compared with the three-dimensional space positions of the corresponding joining points of the three-dimensional virtual image. For example, the comparison may be performed starting from the root node and sequentially from the root node to the child node.

The three-dimensional spatial position difference between each joint point of the object photographing object and the corresponding joint point of the three-dimensional virtual image is less than a preset threshold, or the specified joint point of the photographing object and the corresponding joint point of the three-dimensional virtual image If the 3D spatial position difference between the two is smaller than a preset threshold, it is determined that there is no 3D posture difference between the 3D virtual image and the photographing object. In addition, the 3D posture difference between the photographing object and the 3D virtual image is the angular difference between each bonding point of the photographing object and the corresponding bonding point of the 3D virtual image or the correspondence between the specified bonding point of the photographing object and the 3D virtual image It can be determined through the angular difference between the bonding points. For example, a connection line of some designated bonding points of the photographing object is determined, and a connection line of corresponding bonding points of the three-dimensional virtual image is determined. The 3D posture difference between the photographed object and the 3D virtual image is determined according to the angular difference between the two connecting lines.

In operation 1720, the photographing device 100 outputs a photographing guide according to the determined difference and performs photographing. At this time, the photographing guide is output according to the order, and the posture can be adjusted by guiding the photographing object.

In particular, the shooting object is guided to arrive at a designated position, so that the three-dimensional spatial position of the shooting object is the same as or similar to the three-dimensional spatial position of the three-dimensional virtual image constructed in the background image, and/or the shooting object has a three-dimensional posture is guided to adjust, so that the three-dimensional posture of the photographing object is the same as or similar to the three-dimensional posture of the three-dimensional virtual image. When the shooting object is guided to adjust the three-dimensional space position and three-dimensional posture, after the shooting object is guided to adjust the three-dimensional space position (that is, after reaching the specified position), the shooting object adjusts the three-dimensional posture guided to do

When a photographing object is guided to reach a designated position, the photographing object is guided with respect to a movement direction and a movement distance based on a 3D spatial position difference between the three-dimensional virtual image in the background image and the photographing object.

For example, when a voice guide is used, a voice command, for example, <a distance of 3 m to the front> is transmitted, and the voice guide may be output through the wearable device.

For example, when an image guide is used, at least one of items such as a movement trajectory of a photographing object, a current position and a specified position of the photographing object, and a distance to a specified position may be displayed: object. The image guide may be output through the display screen of the photographing apparatus and/or the wearable device.

Also optionally, the imaging object may be guided via indicator lights of different colors. For example, green light indicates that the photographing object moves forward, and blue light indicates that the photographing object moves to the right. The photographing object may be guided through an indicator light of the photographing apparatus and/or the wearable device.

18 illustrates an example of guiding a photographing object to reach a designated position using an image guide, according to an embodiment. As illustrated in FIG. 18 , information 1840 about a path 1810 to a designated location, a current location 1820, a designated location 1830 of a photographing object, and a distance to the designated location may be displayed for the photographing object.

If there are a plurality of shooting objects, each of the shooting objects may be guided according to the above method to reach a corresponding designated position. For example, each of the photographing objects may be guided according to an order (eg, the photographing object order from left to right, top to bottom, front to rear, from dark clothing color to light clothing color).

19 illustrates an example of guiding a plurality of photographing objects to reach designated positions, according to an embodiment.

Referring to FIG. 19 , movement trajectories 1910 and 1920 of the plurality of photographing objects, current positions 1911 and 1921 of the plurality of photographing objects, corresponding designated positions 1912 and 1922, information on respective distances to corresponding designated positions 1913 , 1923 may be displayed. In addition, different colors may be used to mark paths to corresponding designated locations and designated locations corresponding to current locations of a plurality of photographing objects.

When the photographing object is guided to adjust the posture, the guide is performed based on the 3D posture difference between the photographing object and the 3D virtual image.

For example, when a voice guide is used, a voice command is transmitted, for example, <Raise your left hand by 10 cm, move your left hand backward by 8 cm>.

For example, when an image guide is used, the current posture of the photographing object and the posture of the three-dimensional virtual image are displayed as two images, respectively, or as one image including two overlapping postures, so that the photographing object is It is possible to learn a 3D posture difference between a photographed object and a 3D virtual image.

For example, when an information output method is used for a guide, information on a distance and a direction to be moved for each bonding point may be output. For example, text such as <Move the predetermined bonding point 10 cm to the left and 8 cm backward> may be output.

For example, a color marking method may be used on the output. For example, red indicates that the posture difference is large, yellow indicates that the posture of the photographing object is similar to that of the three-dimensional virtual image, and green indicates that the posture of the photographing object matches the posture of the three-dimensional virtual image. can be used

For example, the indicator lights of different colors may be used to indicate whether the three-dimensional space position of each joint point of the imaging object coincides with the three-dimensional space position of the corresponding joint point of the three-dimensional virtual image.

20 illustrates an example of guiding a photographing object to adjust a posture of the photographing object, according to an exemplary embodiment. As shown in FIG. 20 , a 3D posture difference between a photographing object and a 3D virtual image may be displayed with respect to the photographing object through an image, and the photographing object may be guided through text output 2010 . In addition, the current posture 2020 of the photographing object and the posture 2030 in which the photographing object is to be adjusted under multiple angles can be simultaneously displayed as a front view or a side view, and colors are applied to the current posture of the photographing object and the posture to be adjusted (eg, For example, a color is applied to the current posture of the object to be captured and the posture to which the current posture is to be adjusted based on the joint point). Accordingly, the photographing object can conveniently understand how the posture is adjusted. For example, in some situations, the imaging object may not understand how posture is adjusted with only a single front view or a single side view. For example, a photographing object may need to move the left hand 8 cm backwards, but it may not be determined whether the left hand is moving forward or backward and how far it will be moved in a single front view, but can be accurately provided in a side view. have.

It should be understood that if there are multiple shooting objects, each shooting object is guided to adjust its posture. After all the shooting objects have reached their corresponding designated positions, the guide for each object is started to adjust the posture. Optionally, after a certain shooting object reaches a corresponding designated position, a guide for the object is started to adjust the posture.

21 illustrates an example of guiding a plurality of photographing objects in order to adjust postures of the plurality of photographing objects according to an exemplary embodiment. As shown in FIG. 21 , the difference in posture between one of the photographing objects and the 3D virtual image is displayed through the image, and the photographing objects are guided through text 2110 . Also, images and text corresponding to different shooting objects are marked with different colors.

In an example, the photographing guide is output through the photographing apparatus and/or the wearable device, and thus the photographing object is guided to perform a corresponding adjustment. For example, if the imaging object is in proximity to the imaging device, the imaging object may be guided through a voice command from the imaging device and/or via a rotating display screen. When the photographing object is far from the photographing device, the photographing object is guided through a wearable device (eg, Bluetooth headphones, smart watch, smart glasses, smart bracelet, etc.) that is worn by the photographing object and is connected to the photographing apparatus. For example, when it is necessary to guide a photographing object to perform adjustment, the photographing apparatus may notify the user to use a wearable device for guiding, and display a list of adjacent wearable devices. A user may select and access a corresponding wearable device. Also, the wearable device may initiate a connection request. For example, the wearable device has installed an APP that guides the user to take a picture. This app can make a connection request to the shooting device. It should also be noted that the output of another electronic device (eg, smartphone, tablet computer) carried by the photographing object may be used to guide the photographing object to perform adjustment.

22 illustrates an example of a connected electronic device according to an embodiment. 22 , a message 2210 to select a device to be paired with a three-dimensional virtual image and a list 2220 of electronic devices connectable to the photographing apparatus may be displayed to the user, and the user may select and connect a corresponding electronic device can be established, and the photographing apparatus guides the photographing object through this connection to perform the adjustment.

If there are a plurality of photographing objects, the photographing apparatus may establish a connection with each of a wearable device worn by each object or an electronic device carried by each object, so that the photographing object provides a guide for performing adjustment, a corresponding wearable device or via a corresponding portable electronic device.

23 illustrates another example of displaying on a connected electronic device, according to an embodiment. 23 , a selection box 2310 capable of selecting one of a plurality of three-dimensional virtual images, and a list 2320 of electronic devices connectable to the photographing apparatus, which can be mapped to each three-dimensional virtual image, are displayed for the user. Accordingly, the user may select an electronic device corresponding to each of the plurality of three-dimensional virtual images and establish a connection, thereby guiding the selected photographing object from the plurality of photographing objects and performing adjustment. For example, the user may select the Galaxy S6 as the device to be paired for the first 3D virtual image and may select the Gear VR as the device to be paired with for the second 3D virtual image. Accordingly, the photographing object corresponding to the first 3D virtual image may be provided with a guide through the Galaxy S6, and the photographing object corresponding to the second 3D virtual image may be provided with the guide through the Gear VR.

Preferably, in the process of performing imaging based on the constructed three-dimensional virtual image, the imaging object is focused and photographed at an appropriate time. A procedure of operation 440 of performing imaging based on the configured 3D virtual image will be described with reference to FIG. 24 .

24 is a flowchart illustrating a method of performing imaging based on a configured 3D virtual image, according to another exemplary embodiment.

Referring to FIG. 24 , in operation 2410 , it is determined whether a photographing object matches a 3D virtual image in a background image. For example, if the difference between the 3D virtual image in the background image and the shooting object is less than a preset focusing threshold, or if the shooting object is roughly similar to the 3D virtual image in the background image, the shooting object is a 3D virtual image determined to match the image.

The difference between the 3D virtual image in the background image and the photographing object may include a 3D spatial position difference and a 3D posture difference between the 3D virtual image in the background image and the photographing object. When the comparison is performed, the three-dimensional spatial position difference between the three-dimensional virtual image and the photographing object is smaller than the preset position focusing threshold, and the three-dimensional position difference between the three-dimensional virtual image and the photographing object is the preset posture focusing threshold If less than, it may be determined that the difference between the three-dimensional virtual image and the photographing object is smaller than a preset focusing threshold. Optionally, if the weighted sum of the 3D spatial position difference and the 3D posture difference is less than a preset total focusing threshold, it may be determined that the difference between the 3D virtual image and the photographed object is smaller than the preset focusing threshold.

In operation 2410, if the photographing object matches the 3D virtual image, in operation 2420, the photographing device 100 focuses and photographs the photographing object.

In one example, when the shooting object matches the three-dimensional virtual image, the shooting object is automatically focused and photographed.

In one example, if the photographing object includes a plurality of objects, the procedure of operation 2420 may be performed for each object.

25 illustrates an example of focusing each of a plurality of objects according to an exemplary embodiment. As shown in FIG. 25 , an object matching the corresponding three-dimensional virtual image is focused, and an image including the object is captured. After each object is focused and photographed, the object is extracted from each photographed image, and all extracted objects and background images are combined. For example, referring to FIG. 25 , the photographing device 100 obtains information 2510 indicating that the similarity between the first 3D virtual image and the corresponding photographing object is 90% and the second 3D virtual image and the corresponding photographing object have a similarity of 94% can be displayed. According to this method, each object is focused, images in which different objects are focused are taken respectively, corresponding objects are extracted from the captured images, and all the extracted objects and the background image are combined, so that all objects are clear. An image is acquired. Therefore, if multiple objects are at different scenario depths, the problem that only one object is focused and the other objects are blurred can be solved.

26 illustrates another example of focusing each of a plurality of photographing objects according to an exemplary embodiment. 26 , each of the close-up object and the remote-photographed object is focused to take a corresponding image, and the objects are extracted from the corresponding images, and all the extracted objects and the background image are combined, An image is obtained that includes close-up objects and sharp remote-photographed objects.

The object may be extracted from the corresponding captured image according to various methods. For example, a rough user region may be obtained by mapping the skeleton to the captured image. The area where the object is located is partitioned according to a graph cut algorithm or other partitioning algorithm. The undivided points are background points constituting the background image. For example, an area where the object is located may be detected using a pedestrian detection related algorithm (eg, a DPM model, etc.), and then an area of similar depth is formed using a breadth-first growth algorithm in the detection area where the object is located. It is partitioned using a breadth first growth algorithm. If the precision of the depth information is not sufficient, re-segmentation may be performed using a matting related algorithm. In addition, all extracted objects and background images are combined according to various methods (eg, Poisson function).

In addition, images are captured by focusing each of specific bonding points of the photographing object, and by combining the photographed images, an image including clear body parts of the photographing object located at different scenario depths is obtained. The focused bonding points may be automatically configured according to the posture of the photographing object or may be selected according to a user's motion. For example, when a three-dimensional virtual image is selected, when a three-dimensional spatial position and/or a three-dimensional posture of the three-dimensional virtual image is configured, or when imaging is performed, focused bonding points are selected.

In one embodiment, the three-dimensional virtual image may have a posture used to indicate that the posture of the imaging object for which capture is desired has emerged. Accordingly, when the photographing object is focused in operation 2420, the photographing object is automatically captured, and an image in a posture in which the photographing object is desired to be captured is obtained.

In an embodiment, the photographing posture database may pre-store a series of three-dimensional virtual image postures (ie, posture sequences of the three-dimensional virtual image). Optionally, the posture sequence of the three-dimensional virtual image may be added to the photographing posture database as needed. The user may select a posture desired to be captured from the photographing posture database. Thus, when it is determined that the pose of the imaging object matches a particular pose in front of the desired pose in the pose sequence (i.e., the pose used to predict the appearance of the pose desired to be captured), a method comprising the imaging object of the desired pose to be captured is determined. A photographing object may be focused to capture an image.

27 illustrates an example of capturing a photographing object according to an exemplary embodiment. As shown in FIG. 27 , the shooting posture database may pre-store three-dimensional virtual image posture sequences 2710, 2720, 2730, 2740, and 2750 of basketball shooting, and posture 2740 in FIG. 27 is a posture desired by the user to capture. . Accordingly, a posture capable of predicting the appearance of a shooting object posture desired to be captured can be selected in the basketball shooting posture sequence. For example, in FIG. 27 , posture 2720 may be selected in consideration of a shooting speed and/or a general basketball shooting speed. Therefore, when the posture of the photographing object matches the posture 2720 of FIG. 27, the photographing object is focused, and the photographing object is captured when the photographing object is in the posture 2730 of FIG. 27, including the photographing object in the posture 2740 of FIG. An image may be captured.

In one example, if the photographing object includes a plurality of objects, the procedure of operation 2420 is performed for each object, the objects are extracted from the corresponding photographed image, and all the extracted objects and the background image are combined, where 3 The dimensional virtual image has a posture that predicts the posture of a photographing object for which capture is desired. Thus, an image is obtained that includes objects in different poses that are desired to be captured.

In one example, the three-dimensional virtual image may have a plurality of postures desired to be continuously captured. In operation 2420, when the shooting object continuously matches the 3D virtual image for a preset period, the shooting object is automatically focused and continuous shooting is performed, so that additional actions can be avoided when the continuous shooting is performed have. For example, with the current continuous shooting method, the shooting object is continuously shot for the same time period (eg, the shooting object is shot once every several seconds of time period), and the action of the shooting object used to change the shooting posture is shot can be

In one example, when the shooting object includes a plurality of objects, the object that does not match the three-dimensional virtual image may be removed from the shooting preview image before the object is focused, or the object that does not match the three-dimensional virtual image The object may be removed from the captured image after the object is focused and photographed.

In particular, the shooting object includes a target object (ie, an object to be actually photographed) and a non-target object (eg, a passerby). The target object and the non-target object may be automatically recognized according to whether the photographing object matches the 3D virtual image, and the non-target object may be removed before or after photographing is performed. If the difference between the object and the three-dimensional virtual image is smaller than a preset focusing threshold, the object is recognized as a target object. If the difference between the object and the 3D virtual image is greater than a preset focusing threshold, the object may be recognized as a non-target object. Optionally, only when the difference between the object and the three-dimensional virtual image is greater than a preset deviation threshold, the object is recognized as a non-target object, and the deviation threshold is greater than or significantly greater than the focusing threshold. have.

28 illustrates an example of recognizing a target object and a non-target object according to an embodiment. 28 , a target object and a non-target object may be recognized according to a difference between a photographing object and a 3D virtual image. Target objects and non-target objects are marked separately. Referring to FIG. 28 , in the imaging device 100, the similarity between the first 3D virtual image and the corresponding target object is 90%, the similarity between the second 3D virtual image and the corresponding target object is 94%, and the number of non-target objects is Information 2810 indicating '1' may be displayed.

In an embodiment, according to various methods, if the difference between the object and the three-dimensional virtual image is not less than the focusing threshold, the object is removed from the captured preview image or the captured image. For example, when the non-target object moves, the area previously covered by the non-target object may be recorded so that the area can be filled after the non-target object is removed. If a non-target object is large and does not move for a long period of time (i.e., the gray value or depth value of the corresponding pixel does not change for a long period of time), the area covered by the non-target object can be filled using a selected path match method, or using an image that is similar to the area covered by the non-target object and obtained from the Internet, etc. Optionally, the area covered by the non-target object may not be processed.

29 illustrates an example of removing a non-target object according to an embodiment. As shown in FIG. 29 , each of target objects and non-target objects is determined, and the non-target objects are removed. Referring to FIG. 29 , the photographing device 100 extracts target objects 2910, 2920, and 2930 from a photographing preview image or a photographed image, adds target objects 2910, 2920, and 2930 to the background image 2900, and excludes the non-target object 2940. image can be obtained.

A photographing method according to an embodiment may include: determining a photographing accessory; and configuring the three-dimensional spatial position and/or the three-dimensional posture of the imaging accessory in the background image. The above-described procedure is performed before photographing is performed (eg, the procedure is performed in operation 410 and/or operation 430, or the procedure of operation 430 is performed and the procedure of operation 440 is performed), or the photographing is performed before the procedure of operation 440 is performed. It may be performed while being performed (eg, the procedure is performed in operation 440), or may be performed after photographing is performed (eg, the procedure is performed on a photographed image), but the present embodiment is not limited thereto .

The photographing accessory may be an item that decorates the photographing object of the image, for example, a hat, glasses, a watch, a bag, and the like. The imaging accessory may be displayed to the user through a list, image, three-dimensional model, or the like. The imaging accessory may be determined automatically by the imaging device (eg, the imaging accessory is configured by default on the imaging device), or the imaging accessory may be selected via user input (eg, by clicking or dragging the selected imaging accessory). .

In one example, the three-dimensional spatial position and/or three-dimensional posture of the imaging appendage is roughly constructed in the background image, and then the three-dimensional spatial position and/or the three-dimensional posture of the imaging appendage is precisely adjusted in the background image. . Optionally, the three-dimensional spatial position and/or the three-dimensional pose of the imaging appendage is precisely constructed directly in the background image.

In one example, the three-dimensional spatial position of the imaging appendage is roughly constructed in the background image according to the user action. For example, the selected shooting accessory is moved according to a user moving action (eg, dragging) to a corresponding position in the background image, or the shooting accessory is positioned at a location designated by the user in the background image.

In an example, the three-dimensional spatial position and/or the three-dimensional pose of the imaging appendage determined in the background image is configured automatically. Before imaging is performed, the determined three-dimensional spatial position and/or three-dimensional posture of the imaging accessory in the background image is configured according to the three-dimensional spatial position and/or three-dimensional posture of the three-dimensional virtual image in the background image. For example, the three-dimensional spatial position and/or the three-dimensional pose of the imaging accessory in the background image is configured according to the properties of the imaging accessory, the three-dimensional spatial position and/or the three-dimensional pose of the three-dimensional virtual image in the background image. The properties of the imaging accessory may include a type of the imaging accessory, a body part associated with the imaging accessory, and a binding point bound to the imaging accessory. For example, the three-dimensional spatial position of the imaging accessory in the background image is configured according to the three-dimensional spatial position of the three-dimensional virtual image in the background image. A predetermined or partial bonding point of the three-dimensional virtual image bound with the photographing accessory is determined according to the properties of the photographing accessory (for example, if the photographing accessory is a hat, the photographing accessory is tied with the head of the three-dimensional virtual image, and if the photographing accessory is a bag, the imaging appendage is bound with the hand of the three-dimensional virtual image), and then, the three-dimensional spatial position and/or the three-dimensional posture of the imaging accessory in the background image is further configured according to the three-dimensional spatial position of each of the predetermined or some joining points do. The user may further adjust the three-dimensional spatial position and/or three-dimensional pose of the imaging accessory based on the automatically configured three-dimensional spatial position and/or three-dimensional pose of the imaging accessory.

30 illustrates an example of selecting an imaging accessory and configuring a three-dimensional spatial position of the imaging accessory according to an embodiment. Referring to FIG. 30 , the menu area 130b of the photographing apparatus 100 may display a user interface 3010 for selecting a photographing accessory and a user interface 3020 for selecting a type of the selected photographing accessory. As shown in FIG. 30 , the bag selected by the user is automatically configured in an appropriate position corresponding to a hand joint point of a shooting object in the background image.

When the configuration is performed automatically during the imaging process, the three-dimensional spatial position and/or the three-dimensional pose of the imaging appendage in the background image is configured according to the three-dimensional spatial position and/or the three-dimensional pose of the detected imaging object in the background image . For example, the three-dimensional spatial position and/or the three-dimensional pose of the imaging accessory is configured according to the determined properties of the imaging accessory, the three-dimensional spatial position and/or the three-dimensional pose of the imaging object. For example, the three-dimensional spatial position of the imaging appendage is determined according to the configured three-dimensional spatial position of the imaging object in the background image, and specific or some binding points bound to the imaging accessory are determined according to the properties of the imaging accessory, and the background image The three-dimensional space position and/or the three-dimensional posture of the photographing appendage is additionally configured according to the three-dimensional space position of each of the specific or some bonding points of the photographing object.

In an embodiment, the three-dimensional spatial position and/or the three-dimensional pose of the imaging appendage within the background image may be precisely constructed. The photographing accessory may be configured with angular positions in the horizontal direction, vertical direction, and depth direction, and may further be configured with rotation angles in the horizontal direction, vertical direction and depth direction. The three-dimensional spatial position and/or three-dimensional posture of the imaging appendage in the background image may be constructed according to a method of constructing the three-dimensional spatial position and three-dimensional posture of the three-dimensional virtual image (this method is not repeatedly described). .

31 shows an example of configuring a three-dimensional spatial position of an imaging accessory according to an embodiment. As shown in FIG. 31 , the user selects the three-dimensional spatial location of the selected shooting accessory in the background image, dragging the shooting accessory with one finger 3110, pinching and/or spreading with both fingers 3120, sliding the scroll bar 3130, text It can be configured through user actions such as inputting content corresponding to the box 3140 and voice action 3150. Also, the imaging device 100 may display a notification 3160 indicating the current 3D spatial location of the selected imaging accessory. .

32 illustrates an example of configuring a rotation angle of an imaging accessory according to an embodiment. 32 , the user may configure the rotation angle of the photographing accessory selected in the background image by sliding the scroll bar 3210, inputting content corresponding to the text box 3220, voice operation 3230, and the like. Also, the photographing device 100 may display a notification 3240 of the current rotation angle of the selected photographing accessory.

Further, after the three-dimensional spatial position and/or three-dimensional posture of the imaging accessory in the three-dimensional virtual image is configured in the background, the three-dimensional spatial position and/or the three-dimensional posture of the imaging accessory is configured in the three-dimensional space of the three-dimensional virtual image in the background image It can be adjusted automatically according to spatial position and/or three-dimensional posture. That is, the three-dimensional spatial position and/or the three-dimensional posture of the imaging accessory may be dynamically updated. Thus, the three-dimensional spatial position and/or the three-dimensional posture of the imaging appendage can be changed according to the change of the three-dimensional spatial position and/or the three-dimensional posture of the three-dimensional virtual image, thus giving a dynamic true sense. can be implemented

For example, after the three-dimensional spatial position and/or the three-dimensional posture of the imaging appendage is determined, the relative three-dimensional spatial position and/or relative A three-dimensional posture may be determined. Accordingly, when the three-dimensional virtual image is adjusted, the three-dimensional spatial position and/or the three-dimensional posture of the imaging accessory is adjusted according to the relative three-dimensional space position and/or the relative three-dimensional posture.

In addition, the three-dimensional spatial position and/or the three-dimensional posture of the three-dimensional virtual image in the background image may be automatically adjusted according to the three-dimensional space position and/or the three-dimensional posture of the imaging accessory. That is, the 3D spatial position and/or 3D posture of the 3D virtual image may be dynamically updated. Accordingly, the three-dimensional spatial position and/or the three-dimensional posture of the three-dimensional virtual image may be changed according to the change of the three-dimensional spatial position and/or the three-dimensional posture of the imaging accessory, and as a result, a dynamic real sense is realized.

Also, color can be applied to an imaging accessory. For example, physical properties of the imaging accessory (eg, weight, surface tension, elasticity) may be adjusted to make the imaging accessory more realistic.

A photographing method according to an embodiment is disclosed in FIG. 33 . According to an embodiment, the shooting object is guided to perform an adjustment based on the selected virtual image. This method may be implemented through a photographing device or may be implemented through a computer program. For example, the method may be implemented with a camera installed in the photographing apparatus or may be implemented through a function program of the photographing apparatus operating system.

As shown in FIG. 33 , in operation 3310 , the photographing device 100 may configure a virtual image used to represent a photographing object in a background image. The virtual image may be a three-dimensional or two-dimensional person model with a posture, and the present embodiment is not limited thereto. For a detailed method of configuring the virtual image used to represent the photographing object in the background image, refer to the detailed implementation method (not repeatedly described) of operation 430 of configuring the 3D virtual image in the background image.

In operation 3320, the photographing device 100 may determine a difference between the virtual image in the background image and the photographing object.

The photographing apparatus may automatically detect the photographing object. The detailed detection method may refer to the detailed implementation method for automatically detecting a photographed object. For example, the photographing object may be detected based on the posture of the virtual image in the background image, or may be detected based on spatial location information provided by the wearable device. Optionally, the object selected by the user in the background image is regarded as a shooting object, and then the selected shooting object is tracked as a target. The tracking method will not be repeatedly described.

In one example, the difference between the virtual image and the photographing object in the background image may include a spatial position difference and/or posture difference between the virtual image and the photographing object in the background image. If the determined virtual image is a three-dimensional virtual image, and the background image has depth information, the difference between the virtual image in the background image and the photographed image is a three-dimensional spatial position difference between the virtual image and the photographing object in the background image and/or 3 Dimensional posture differences may be included.

In particular, the method for determining the difference between the 3D virtual image in the background image and the photographing object may refer to the above-described method, and the above-described method will not be repeatedly described. For example, the posture difference between the photographing object and the virtual image is determined by comparing the spatial position of the bonding point of the photographing object with the spatial position of the corresponding bonding point of the virtual image.

When the determined virtual image is a two-dimensional image, the difference between the virtual image and the photographing object in the background image may include a two-dimensional spatial position difference and/or a two-dimensional posture difference between the photographing object and the virtual image. An image recognition method may be used to determine a two-dimensional spatial position difference and/or a two-dimensional posture difference.

In operation 3330, the photographing device 100 may output a photographing guide based on the determined difference.

The difference between the virtual image and the shooting object in the background image includes only the spatial position difference between the shooting object and the virtual image, and the shooting object is guided to adjust the spatial position of the shooting object based on the determined spatial position difference, It should be noted that it is not guided to adjust the posture of the object. Therefore, the spatial position of the photographing object is the same as or similar to the spatial position of the virtual image in the background image. If the difference between the virtual image and the photographing object in the background image includes only the difference in posture between the photographing object and the virtual image, the photographing object is guided to adjust the posture of the photographing object based on the determined difference in posture, and the space of the photographing object It is not guided to adjust the position. Accordingly, the posture of the photographing object is the same as or similar to the posture of the virtual image. If the difference between the virtual image and the photographing object in the background image includes the spatial position difference and the posture difference between the photographing object and the virtual image, the photographing object is guided to adjust the spatial position of the photographing object based on the determined spatial position difference and is guided to adjust the posture of the photographing object based on the determined posture difference. Therefore, each of the spatial position and posture of the photographing object is the same as or similar to that of the virtual image. For example, the shooting object is first guided to adjust the spatial position, and then guided to adjust the posture after the shooting object reaches the corresponding spatial position.

The detailed guide method may refer to the detailed implementation method described above (eg, a photographing guide is output through a photographing apparatus and/or a wearable device), and will not be repeated herein.

Another photographing method according to an embodiment is disclosed in FIGS. 34 to 37 . According to the method, a photographing object is focused based on the posture of the photographing object. The method may be implemented through a photographing device or may be implemented through a computer program. For example, the method may be implemented through a camera installed in the photographing apparatus or may be implemented through a function program of the photographing apparatus operating system.

34 is a flowchart illustrating a photographing method according to another exemplary embodiment.

As shown in FIG. 34 , in operation 3410 , the photographing device 100 may detect the posture of the photographing object in the background image. The posture of the photographing object in the background image may be detected through various existing image recognition methods (not described again to avoid repetition).

In operation 3420, the imaging device 100 may determine whether the detected posture of the photographing object matches the focusing posture. The focusing posture is a pre-configured and desired shooting posture of the virtual image representing the imaging object (eg, the posture of the virtual image pre-selected by the user or the posture after the posture of the pre-selected virtual image is configured) or a pre- in the shooting posture database It may be any configured photographing posture or an optimal photographing posture of the photographing object.

35 illustrates an example of determining whether a posture of a photographing object matches a focusing posture, according to an exemplary embodiment. 35 , the focusing posture is an arbitrary photographing posture in the photographing posture database. If the photographing object includes a plurality of objects, it is determined whether each object matches any photographing posture in the photographing posture database. If the posture of the object matches any shooting posture in the photographing posture database, the posture of the object matches the focusing posture, and it is determined that the object is an object to be focused. If the posture of the object does not match any photographing posture in the photographing posture database, the posture of the object does not match the focusing posture, and it is determined that the object is not an object to be focused. Also, the photographing device 100 may display a notification 3510 indicating an object to be focused and an object not to be focused.

For a method of determining whether the detected posture of the photographing object matches the focusing posture, the detailed implementation method for determining a difference between a three-dimensional virtual image in a background image and a photographing object may be referred to. Optionally, the difference between the two-dimensional or three-dimensional posture of the photographing object and the focusing posture may be determined according to another method. For example, when the difference between the posture of the photographing object and the focusing posture is less than a preset focusing threshold, or when the posture of the photographing object is approximately similar to the focusing posture, it is determined that the posture of the photographing object matches the focusing posture do.

If the difference between the detected posture and the focusing posture of the photographing object is less than the first focusing threshold, it is determined that the detected posture of the photographing object coincides with the focusing posture. If the difference between the detected posture of the photographing object and the focusing posture is greater than the second focusing threshold, it is determined that the detected posture of the photographing object does not coincide with the focusing posture. The second focusing threshold may be equal to, greater than, or less than the first focusing threshold. For example, the three-dimensional spatial position difference between each bonding point of the photographing object and the bonding point corresponding to the focusing posture is less than a preset position threshold, or between the specified bonding point of the photographing object and the bonding point corresponding to the focusing posture. If the three-dimensional spatial position difference of is less than a preset position threshold, it is determined that the posture of the photographing object coincides with the focusing posture. Optionally, an angular difference between each bonding point of the photographing object and a bonding point corresponding to the focusing posture is less than a preset angle threshold, or an angular difference between a designated bonding point of the photographing object and a bonding point corresponding to the focusing posture is If it is smaller than the preset angle threshold, it is determined that the posture of the photographing object coincides with the focusing posture.

Referring back to FIG. 34 , in operation 3430 , if the posture of the photographing object matches the focusing posture, the photographing device 100 may focus and photograph the photographing object.

In one example, when the detected posture of the photographing object matches the focusing posture, the photographing object may be automatically focused and photographed. That is, in a situation in which the user does not need to perform any action, the photographing object is automatically focused and photographed.

In one example, when the photographing object includes a plurality of objects, the procedure of operation 3430 may be performed for each object. After the procedure of operation 3430 is performed for each object, an object may be extracted from a corresponding captured image, and all the extracted objects and background images are combined. According to this method, each object is individually focused, an image in which the object is focused can be taken, a corresponding object is extracted from the shot image, and all extracted objects and background images are combined, so that all objects are A clear image can be obtained. In particular, for the method of extracting each object from the corresponding image and combining all the extracted images and the background image, the detailed implementation method may be referred to above. Detailed implementation methods are not repeatedly described.

In one example, the focusing posture is used to predict the appearance of an imaging object posture desired to be captured. Accordingly, in operation 3430 , the imaging object is focused and automatically captured, thus obtaining an image of the imaging object in the desired pose to capture. For example, the desired posture to capture may be the best photographing posture of the photographing object, and the optimal photographing posture of the photographing object may be the posture found by the photographing object at a best location, a peak, etc. have. For example, a posture desired to be captured may be a desired posture selected by a user from several posture sequences stored in a photographing posture database. The focusing posture is a posture used to predict the appearance of a posture of a capturing object desired to be captured (eg, a specific capturing posture before the posture desired to be captured in the capturing posture sequence). When the detected posture of the shooting object matches any posture used for predicting the appearance of the posture of the shooting object desired to be captured, the shooting object is focused, and an image of the shooting object having the posture of the shooting object desired to be captured is displayed. automatically captured. In one example, if the photographing object includes a plurality of objects, the procedure of operation 3430 is performed for each object, an object is extracted from a corresponding photographed image, and all extracted object images and background images are combined, where: The focusing posture is used to predict the appearance of an imaging object posture desired to be captured. Thus, an image is obtained in which each object is in the desired position to be captured.

36 illustrates another example of capturing a photographed object according to an embodiment. As shown in FIG. 36 , the desired posture to capture is the best photographing posture of each of the plurality of objects, and the best photographing posture of the photographing object is the posture in which the photographing object is located at the normal (highest height). Therefore, when the posture of each object matches the posture used to predict the appearance of the posture taken by the photographing object located at the top, the object is focused. Accordingly, an object image in the posture of the object located at the top is obtained. Referring to FIG. 36 , in an image 3610 including a posture in which the first object 3601 is at the highest position, the first object 3601 is focused, and in an image 3620 including a posture in which the second object 3602 is at the highest position, the second The object 3602 may be focused, and the third object 3603 may be focused in an image 3630 including a posture in which the third object 3603 is at the highest position.

In one example, the focusing posture may be used to indicate a series of postures of a photographing object that is desired to be continuously imaged. In operation 3410, the posture of the photographing object is continuously detected from the background image. In operation 3430, if the detected postures of the shooting object consecutively coincide with the focusing postures for the time period, the shooting object is continuously photographed to avoid additional action of the shooting object being photographed when shooting is continuously performed .

Also, in one example, if the photographing object is continuously detected in the background image in operation 3410, the auxiliary photographing method determines whether the posture of the photographing object matches for a preset time period after the procedure of operation 3410 is performed and, when the postures of the photographing object match for a preset time period, the method may further include focusing the photographing object and continuously photographing the photographing object. That is, when it is determined that the photographing object does not change its posture (ie, the still posture of the photographing object), the photographing object is automatically focused and photographed. Compared with the existing continuous shooting method (ie, shooting is performed every time period), the present method can reduce shooting of an additional action.

In an example, the auxiliary imaging method may further include configuring the spatial position and/or posture of the imaging accessory in the background image. For a detailed implementation method for determining and configuring an imaging accessory, reference may be made to the above-described method (not repeatedly described).

In one example, according to an embodiment, when a photographing object includes a plurality of objects, the procedure of operation 3430 includes removing an object having a posture that does not match the focusing posture from the capture preview image before the object is focused and photographed , or removing an object having a posture that does not match the focusing posture from the captured image after the object is captured. In particular, the shooting object includes a target object (ie, an object to be actually photographed) and a non-target object (eg, a passerby). The target object and the non-target object may be automatically recognized according to whether the photographing object matches the focusing posture. Non-target objects are removed before or after imaging is performed. For a detailed implementation method of removing an object having a posture that does not match the focusing posture, the detailed implementation method (not repeatedly described) may be referred to.

37 illustrates another example of removing a non-target object according to an embodiment. 37 , the focusing posture is an arbitrary photographing posture in the photographing posture database, and the photographing object includes a plurality of objects. Whether each object is a target object or a non-target object may be determined, and the target object and the non-target object to be removed may be displayed to the user.

A photographing method when the photographing object is a single object is described in detail.

38 is a flowchart illustrating a photographing method according to an exemplary embodiment. This method is used for a single shooting object.

As shown in FIG. 38 , in operation 3810 , the photographing device 100 starts an operation of a camera application. The camera application may be initiated by a user action starting the camera application. For example, if it is detected that the user clicks on the camera icon, the camera application is started. Optionally, when a voice command to launch the camera application is detected, the camera application may be launched.

In operation 3820, the imaging device 100 may acquire a background image having depth information. That is, a shooting preview image (having depth information) captured in real time through the depth camera is displayed to the user through the screen of the imaging device.

In operation 3830, the photographing device 100 may select a 3D virtual image used to represent a photographing object.

According to a user action for the <Select 3D Virtual Image> menu item in the menu bar, 3D virtual images in the photographing posture database are displayed on the side of the screen for the user to select. The user can select the three-dimensional virtual image by clicking on it.

In addition, the user may additionally select a shooting scenario template including a 3D virtual image and a shooting accessory (eg, the sun). The relationship (eg, positional relationship) between the imaging appendage and the three-dimensional virtual image is defined by a pre-configured imaging scenario template.

In operation 3840, the imaging device 100 may configure the 3D spatial position and 3D posture of the 3D virtual image in the background image.

In one example, the three-dimensional spatial position of the three-dimensional virtual image may be first roughly constructed and then further adjusted.

For example, the user may click on the selected three-dimensional virtual image and drag the image to a corresponding position in the background image displayed on the other side of the screen. Accordingly, the selected three-dimensional virtual image is roughly constructed within the background image. Preferably, an area suitable for positioning a three-dimensional virtual image and an area not suitable for positioning the three-dimensional virtual image may be displayed in the background image. Optionally, when positioning the 3D virtual image in an area that is not suitable for positioning the 3D virtual image, the user may be notified of this.

For example, if a shooting scenario template comprising a three-dimensional virtual image and an imaging appendage is selected in block S403, the three-dimensional spatial position and the three-dimensional posture of the three-dimensional virtual image may be automatically constructed based on the imaging appendage in the background image. can

After the three-dimensional spatial position of the three-dimensional virtual image is roughly constructed, an interface used to precisely construct the three-dimensional space position of the three-dimensional virtual image is displayed on the other side of the screen. For example, a scroll bar or text box used to adjust the positions of the three-dimensional virtual image in the horizontal direction, the vertical direction and the depth direction is displayed on the interface. The user may precisely configure the 3D spatial position of the 3D virtual image by sliding the scroll bar or inputting coordinates corresponding to the text box.

After the user determines that the procedure for constructing the three-dimensional space position is finished, an interface used to construct the three-dimensional posture of the three-dimensional virtual image is displayed on the side of the screen. For example, a schematic diagram illustrating a posture of a three-dimensional virtual image expressed through a joining point of the three-dimensional virtual image may be displayed on the interface, and the schematic diagram may include a front view or a side view. The user can configure the three-dimensional posture of the three-dimensional virtual image by clicking the joint point to be constructed and dragging the joint point to the corresponding position. In addition, the user may further configure the rotation angle of the 3D virtual image by clicking the 3D virtual image and dragging to rotate the 3D virtual image.

Preferably, the user may further select an imaging accessory (eg, a hat). The three-dimensional spatial position and three-dimensional posture of the imaging appendage are automatically configured according to the properties of the imaging appendage (the joining points of the three-dimensional virtual image are combined), the configured three-dimensional space position of the three-dimensional virtual image, and the configured three-dimensional posture. The user may further adjust the three-dimensional spatial position and three-dimensional posture of the imaging accessory.

In operation 3850, the photographing device 100 may output a photographing guide based on a difference between the 3D virtual image in the background image and the photographing object. In particular, the shooting guide is output based on the 3D spatial position difference and the 3D posture difference between the shooting object and the 3D virtual image.

For example, the photographing object may be first guided to arrive at a location where the three-dimensional virtual image is positioned based on the three-dimensional spatial position difference, and then guided to adjust the three-dimensional posture based on the three-dimensional posture difference. A voice guide may be output through the photographing device. Optionally, the image guide may be output through a wearable device worn by the photographing object and paired with the photographing apparatus.

In operation 3860, the imaging device 100 may remove the non-target object from the background image. That is, the non-target object (ie, a passerby) is removed from the shot preview image and is not displayed in the shot image.

The procedures of operation 3850 and operation 3860 may be performed simultaneously. That is, the photographing apparatus may simultaneously guide the photographing object and remove the non-target object.

In operation 3870, when the photographing object matches the 3D virtual image in the background image, the photographing device 100 may automatically focus and photograph the photographing object. For example, the three-dimensional spatial position difference between the imaging object and the three-dimensional virtual image is smaller than a preset position focusing threshold, and the three-dimensional position difference between the imaging object and the three-dimensional virtual image is smaller than the preset posture focusing threshold. , it is determined that the photographing object matches the three-dimensional virtual image, and the photographing object is automatically focused and photographed.

A photographing method when the photographing object includes a plurality of objects will be described in detail below.

39 is a flowchart illustrating a photographing method according to an exemplary embodiment. This method is used for multiple shooting objects.

As shown in FIG. 39 , in operation 3910 , the photographing device 100 may initiate a camera application. The camera application may be initiated by a user action starting the camera application. For example, if it is detected that the user clicks on the camera icon, the camera application is started. Optionally, when a voice command to launch the camera application is detected, the camera application is started.

In operation 3920, the photographing device 100 may acquire a background image. That is, a shooting preview image (having depth information) captured in real time through the depth camera is displayed to the user through the screen of the imaging device.

In operation 3930, the photographing device 100 may select virtual images used to represent a plurality of photographing objects.

According to a user action on the <select virtual images> menu item in the menu bar, virtual images in the photographing posture database are displayed on the side of the screen for the user to select. The user may select a virtual image for each object or a combination of multiple virtual images.

In operation 3940, the photographing device 100 may output photographing guides based on virtual images corresponding to photographing objects in the background image and differences between the photographing objects. That is, each of the photographing guides is output based on a difference in posture between each photographing object and a virtual image corresponding to the photographing object. For example, shooting objects are continuously guided in order (from left to right) through voice commands. Optionally, each of the audio or image guides is transmitted to the wearable device worn by the corresponding photographing objects, so that each photographing object can be adjusted according to the guide from the corresponding wearable device.

In operation 3950 , when the posture of the photographing object matches the posture of the corresponding virtual image with respect to each object, the photographing device 100 may automatically focus the photographing object to take a photograph. That is, for each object, when the detected posture of the photographing object matches the posture of the corresponding virtual image, the photographing object is automatically focused and photographed, so that an image in which each photographing object is focused can be obtained.

In operation 3960, the photographing device 100 may extract each photographing object from a photographed image corresponding to the photographing object, and combine all the extracted photographing objects with a background image. Accordingly, a clear image of all the photographing objects is obtained.

According to the photographing method and apparatus provided according to the embodiments, the 3D virtual image is configured in the background image with depth information, and thus the user can know the photographing effect in advance. In addition, the shooting object is guided to perform adjustment according to the difference between the three-dimensional virtual image and the shooting object in the background image, so that the shooting object matches the virtual image. Also, the photographing object may be focused and photographed based on the posture of the photographing object.

The method of operating a photographing apparatus according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also included in the scope of the present invention. belongs to

Claims (20)

In the photographing device,
a photographing unit for photographing an image; and
Including a control unit connected to the photographing unit,
The control unit is
Acquire a background image with depth information,
disposing a three-dimensional virtual image used to represent a target object to be photographed on the background image having the depth information,
while the capturing unit is positioned to capture a scene corresponding to the background image, determining a difference between the three-dimensional virtual image of the background image and the target object in the live preview image of the capturing unit,
and controlling the photographing unit to automatically photograph the target object when the difference becomes smaller than a predetermined threshold. The method of claim 1,
The control unit is
A photographing apparatus for determining a three-dimensional virtual space position and/or a three-dimensional posture of the three-dimensional virtual image. 3. The method of claim 2,
The control unit is
determine a photographic appendage, and
and further configuring a three-dimensional spatial position and/or a three-dimensional posture of the photographing accessory in the background image. 4. The method of claim 3,
The control unit is
disposing the three-dimensional virtual image based on the shooting appendage of the three-dimensional virtual image in the background image;
and the relationship between the three-dimensional virtual image and the shooting accessory of the three-dimensional virtual image is defined by a pre-configured shooting scenario template. 4. The method of claim 3,
The control unit is
A photographing apparatus for configuring a three-dimensional posture of the three-dimensional virtual image by arranging three-dimensional spatial positions of joint points of the three-dimensional virtual image. According to claim 1,
The control unit is
and outputting a photographing guide indicating the difference through the photographing apparatus and/or a wearable device based on the determined difference. 7. The method of claim 6,
The control unit is
The target object is detected based on the 3D posture of the 3D virtual image, or based on 3D spatial location information provided by the wearable device, or target tracking for the target object selected by the user (target A photographing device that detects based on tracking). 7. The method of claim 6,
The difference between the target object and the 3D virtual image includes a 3D spatial position difference and/or a 3D posture difference between the target object and the 3D virtual image. 9. The method of claim 8,
The control unit is
determining the 3D posture difference between the target object and the 3D virtual image by comparing the 3D spatial position of the bonding point of the target object with the 3D spatial position of the corresponding bonding point of the 3D virtual image , shooting device. According to claim 1,
The control unit is
detecting the posture of the target object in the background image,
determining whether the posture of the target object matches the focusing posture,
When the detected posture of the target object matches the focusing posture, controlling the capturing unit to focus and photograph the target object. 11. The method of claim 10,
wherein the focusing posture is a pre-configured posture used to indicate a desired shooting posture of the photographing object, or an arbitrary shooting posture in a pre-configured photographing posture database. 11. The method of claim 10,
The control unit is
determining an expected pose of the three-dimensional virtual image used to predict the appearance of the target object pose that is desired to be captured;
and controlling the photographing unit to automatically focus and capture the target object when detecting that the target object assumes the expected posture. 11. The method of claim 10,
The control unit is
determining multiple poses of the three-dimensional virtual image used to represent a series of target object poses that are desired to be imaged successively;
When the target object continuously matches the 3D virtual image in the background image for a preset time, the photographing apparatus controls the photographing unit to automatically focus and continuously photograph the target object. According to claim 1,
The target object includes multiple objects,
The control unit is
Before or after the target object is focused and photographed, an object that does not match the 3D virtual image is removed. In the method of operating a photographing device,
acquiring a background image with depth information;
arranging a three-dimensional virtual image used to represent a target object to be photographed on the background image having the depth information;
while the imaging device is positioned to capture a scene corresponding to the background image, determining a difference between the three-dimensional virtual image of the background image and the target object in a live preview image of the imaging device;
and automatically capturing the target object when the difference becomes smaller than a predetermined threshold. 16. The method of claim 15,
and determining a three-dimensional virtual space position and/or a three-dimensional posture of the three-dimensional virtual image. 16. The method of claim 15,
determining a difference between the target object and the three-dimensional virtual image; and
and outputting a photographing guide indicating the difference based on the determined difference through the photographing apparatus and/or a wearable device. 16. The method of claim 15,
detecting the posture of the target object in the background image;
determining whether the posture of the target object matches the focusing posture, and
and focusing and photographing the target object when the detected posture of the target object matches the focusing posture. 16. The method of claim 15,
determining an expected pose of the three-dimensional virtual image used to predict the appearance of the target object pose that is desired to be captured; and
and automatically focusing and photographing the target object when detecting that the target object assumes the expected posture. A computer-readable recording medium in which a program for executing the method according to claim 15 is recorded.

Images