TWI691205B - Method for processing augmented reality image - Google Patents

Abstract

A method for processing augmented reality image is disclosed and includes following steps: taking a photo through a camera of an electronic device; identifying a target object from the photo through an identification algorithm; performing a background-removing procedure to the photo for generating an object image that corresponds to the target object through an image editing software; transforming the object image into an augmented reality (AR) image by the electronic device; displaying the AR image through a displaying unit of the electronic device; and, moving the AR image displayed on the electronic device to aim at an adjusted target in the environment, so as to simulate the appearance of the adjusted target after it is applied with the target object thereon.

Description

Augmented reality image processing method

The invention relates to an image processing method, in particular to an augmented reality image processing method.

With the prevalence of the Internet, many online shopping platforms have appeared on the market, making it convenient for consumers to shop at home.

However, the biggest problem with online shopping is that consumers cannot try or try on them before buying, so they often buy items that are not suitable for them. This also constitutes the main reason for the high return rate of online shopping.

As mentioned above, the same problem also appears in print publications such as magazines and catalogs. When consumers see an item of interest in the above-mentioned print publication, they often give up the idea of further searching and buying this item because they cannot try or try on it.

In addition, even if consumers see physical items around their lives, they may still be unable to obtain the physical items (such as clothes worn by strange passers-by or tiles on buildings), or difficult to obtain (such as watches displayed in the window or Shoes) and other reasons, but hindered the purchase behavior of consumers.

In view of the above-mentioned problems, there is a need to provide a novel technology on the market that can help consumers instantly generate items of interest as augmented actual images, so that consumers can apply items to the target to be modified, thereby determining consumption The purchase intention of the person.

The main objective of the present invention is to provide an augmented reality image processing method that can facilitate the user to generate augmented reality images of a specific item, and the generated augmented reality images are used to simulate the use of this item Appearance.

In order to achieve the above object, the augmented reality image processing method of the present invention mainly includes the following steps: taking a photo through a camera lens of an electronic device; identifying a target object from the photo through a recognition algorithm; The editing software performs a de-backing process on the photo to generate an object image corresponding to the target object; converts the object image into an augmented reality image; displays the augmented reality image on a display unit of the electronic device; and , Moving the electronic device and aligning the augmented reality image with a modified target in the environment, thereby simulating that the target object is used to modify the appearance of the modified target.

The technical effect achieved by the present invention relative to the related art is that, through an electronic device, images of physical objects around the life or items displayed on web pages, magazines, and catalogs are automatically converted into augmented actual images, thereby facilitating users This item is directly applied to the target to be modified to confirm in advance the appearance of this target after using this item.

1: Electronic device

11: Camera lens

12: Processing unit

13: Storage unit

131: Identification algorithm

132: Image editing software

133: 3D database

14: display unit

15: input unit

2: user

3: computer screen

31: target object

4: Photos

5: AR image

6: AR image after editing

7: Modify the target

S10~S24, S30~S44: processing steps

FIG. 1 is a first specific embodiment of a block diagram of an electronic device of the present invention.

FIG. 2 is a first specific embodiment of the image processing flowchart of the present invention.

FIG. 3A is a schematic diagram of the first operation of the AR video of the present invention.

FIG. 3B is a schematic diagram of the second operation of the AR image of the present invention.

FIG. 3C is a schematic diagram of the third operation of the AR video of the present invention.

3D is a schematic diagram of the fourth operation of the AR image of the present invention.

FIG. 4 is a first specific embodiment of the image processing flowchart of the present invention.

The following is a detailed description of a preferred embodiment of the present invention with reference to the drawings.

The present invention discloses an Augmented Reality (AR) image processing method (hereinafter will be referred to as a processing method in the specification for short). The processing method can help a user to quickly generate an AR image, by which the user can Align the AR image with the modified target (person, object or position), thereby simulating the modified appearance of this modified target in advance.

Referring to FIG. 1, it is a first specific embodiment of a block diagram of an electronic device of the present invention. The processing method of the present invention is mainly applied to the electronic device 1 shown in FIG. 1. Specifically, the user can operate the electronic device 1 to instantly convert a target object into an AR image, and apply the AR image to the modification target In order to check in advance the appearance of this modified target after the target object is actually applied. In this way, it is helpful for the user to decide whether to purchase and use the target object.

As shown in FIG. 1, the electronic device 1 mainly includes a camera lens 11, a processing unit 12, a storage unit 13, a display unit 14, and an input unit 15, wherein the camera lens 11, the processing unit 12, the storage unit 13, the display unit 14 and The input units 15 are electrically connected to each other through bus bars. To facilitate user operation, the electronic device 1 can be a handheld electronic device, such as a smart phone or a tablet computer, as an example, but it is not limited.

In an embodiment, the electronic device 1 can mainly take pictures through the camera lens 11 and process the taken pictures through the processing unit 12. In this embodiment, the storage unit 13 stores at least the identification algorithm 131 and the image editing software 132. The processing unit 12 can mainly obtain the camera mirror After the photo taken by the head 11, the recognition algorithm 131 is executed to process the photo by the recognition algorithm 131, so as to recognize the image of the target object from the photo, and further convert the image of the target object into the AR image.

Specifically, the user can operate the electronic device 1 to aim the camera lens 11 at a target object of interest (such as physical objects, magazine or webpage content, etc.), and take a photo, wherein the taken photo must include at least all Describe the target object and background information other than the target object.

Then, after obtaining the photo, the processing unit 12 can identify the part belonging to the target object from the photo through the recognition algorithm 131, and perform a de-backing process on the photo through the image editing software 132 to generate a corresponding to the target object Object image. In this embodiment, the object image generated by the de-backing process only includes the image of the target object in the photo, but does not include the back-view information.

Then, the processing unit 12 may perform further conversion processing on the object image to convert the object image into an AR image. In this embodiment, since the object image only includes the target object, the AR image generated by the conversion process is the AR image of the target object.

For example, if the user takes a photo in the store through the camera lens 11 of the electronic device 1, the photo contains the target object (such as a watch) and background information (such as a watch box, table, and cabinet), then the electronic device 1 After the photo is obtained, the image of the watch can be recognized from the photo by the recognition algorithm 131, and the background information such as the watch box, table, and cabinet in the photo is removed by the image editing software 132 to complete the backing process and generate the object image. Moreover, the electronic device 1 may further perform a conversion process to convert the object image into the AR image of the watch. After the AR image of the watch is generated, the user can move the electronic device 1 by himself to apply the AR image to his wrist, and check his appearance after wearing the watch.

In an embodiment, the image editing software 132 may be, for example, Photoshop editing software developed by Adole, but it is not limited. The image editing software 132 can perform the editing operation required by the user on the AR image after the AR image is generated, in addition to performing the de-keying process.

Specifically, after the AR image is generated, the processing unit 12 can automatically perform related editing operations on the AR image through the image editing software 132, or accept an external operation of the user through the input unit 15, so that the user can manually perform the operation on the AR image The required editing action. In this embodiment, the editing action may be, for example, enlarging, reducing, stretching, deforming, layer replacement, or placing text, etc., but it is not limited thereto. In the present invention, the user can edit the AR image according to the characteristics (shape, size, position, angle, etc.) of the modification target to be applied, so that the AR image can be directly applied to the modification target after the editing is completed.

The electronic device 1 can display the AR image through the display unit 14, whereby the user can move the electronic device 1 to align the AR image displayed on the display unit 14 with a modification target (eg, person, pet) in the environment Objects or walls, windows, floors, etc.) to simulate the appearance of the modified object after the target object represented by the AR image is actually applied.

In an embodiment, the display unit 14 and the input unit 15 may be separate components (for example, the display unit 14 is a screen, and the input unit 15 is a keyboard or a touchpad), or may be integrated into a single component (for example, touch Screen), not limited.

Please also refer to FIG. 2, which is a first specific embodiment of the image processing flowchart of the present invention. FIG. 2 discloses various steps of the processing method of the present invention, and these steps are mainly implemented by the electronic device 1 shown in FIG. 1.

As shown in FIG. 2, to execute the processing method of the present invention, the user first operates the electronic device 1, aligns the camera lens 11 of the electronic device 1 with the target object of interest (step S10 ), and takes a photo (step S12), wherein the photo contains at least an image of the target object.

In an embodiment, the target object may be an object that the user cannot directly obtain (such as an object published in a magazine or an object displayed on a computer screen). In this embodiment, the user can directly operate the electronic device 1 to take a photo including the target object, and in a subsequent step, the processing unit 12 extracts the image of the target object from the entire photo.

In another embodiment, the target object may be a physical object (such as clothes, jewelry, or materials) that the user can directly obtain. In this embodiment, the user can first place the target object on a target background (for example, a completely white background), and then take the above-mentioned photos through the electronic device 1. In this embodiment, the photograph taken by the electronic device 1 only includes images of the target object and the target background. In this way, the processing unit 12 can use fewer resources in the subsequent procedures to complete the identification procedure and the de-encryption procedure, thereby improving processing efficiency and avoiding waste of resources.

After step S12, the processing unit 12 of the electronic device 1 executes the recognition algorithm 131 to recognize the target object from the photo through the recognition algorithm 131 (step S14).

It is worth mentioning that the identification algorithm 131 can be an artificial intelligence (Artificial Intelligence, AI) algorithm, which uses pre-trained artificial intelligence to identify the photo to identify the target object from the photo .

Specifically, the provider of the recognition algorithm 131 may train the artificial intelligence in advance, so that the artificial intelligence recognizes a plurality of target objects (such as clothes, pants, shoes, watches, accessories, electric lights, ceiling fans, tiles, curtains, etc.) ) Appearance (including image, color, shape and other recognizable features). Therefore, in the above step S14, the artificial intelligence can compare the image, color, The shape and other characteristics are analyzed to distinguish the target object and the background information other than the target object from the photo.

After the above step S14, the processing unit 12 can virtually divide the received photos into target objects to be retained and background information that does not need to be retained. Therefore, the processing unit 12 can further execute the image editing software 132 and use the image editing software 132 Perform a debounce procedure on the photo to generate an object image corresponding to the target object (step S16).

Specifically, in the above step S16, the processing unit 12 mainly deletes the image recognized by the recognition algorithm 131 as background information in the photo through the image editing software 132, and retains the image recognized by the recognition algorithm 131 as the target object, And the object image is generated according to the image of the target object, thereby achieving the purpose of removing the back.

Next, the processing unit 12 further performs conversion processing on the generated object image to convert the object image into an AR image (step S18), and displays the AR image through the display unit 14 of the electronic device 1 (S22).

In this embodiment, the AR image refers to an image that can coexist with the image around the reality, which means that when the electronic device 1 activates the camera lens 11 and loads the AR image, the display unit 14 can display the captured image of the camera lens 11 The captured real-time image can display the loaded AR image at the same time. In this way, the user can operate the electronic device 1 and turn on the camera lens 11, and move the electronic device 1 to align the AR image displayed by the display unit 14 with a modification target in the environment (step S24), thereby simulating the The modification target actually applies the appearance of the target object.

It is worth mentioning that the AR image generated in step S18 may not be able to meet the user's needs (for example, the size is too small or deformed) due to the direction, angle, and distance when taking pictures. Therefore, after the step S18, the processing unit 12 may further use the image editing software 132 to The generated AR image is edited (step S20). In this embodiment, the processing unit 12 mainly displays the edited AR image on the display unit 14 in the above step S22, thereby facilitating the user to more appropriately apply the AR image displayed on the display unit 14 to the modification target on.

In one embodiment, the processing unit 12 can automatically perform related editing operations on the AR image through the image editing software 132 (for example, automatically correct the skewed image), or accept an external operation of the user through the input unit 15 to The user manually operates the image editing software 132 to perform required editing operations on the AR image. In this embodiment, the image editing software 132 may be, for example, Photoshop software, and the editing operation may be, for example, zooming in, zooming out, stretching, deforming, layer replacement, or placing text, etc., but not limited thereto.

Please refer to FIG. 3A to FIG. 3D for the first to fourth schematic diagrams of the AR image of the present invention.

As shown in FIG. 3A, an image of a target object 31 is displayed on the computer screen 3. In the embodiment of FIG. 3A, the target object 31 is a tie as an example. The user 2 holds the electronic device 1 and aligns the camera lens 11 of the electronic device 1 with the computer screen 3 to take a photo 4, and the photo 4 contains at least the image of the target object 31.

Next, as shown in FIG. 3B, after the electronic device 1 obtains the photo 4, the processing unit 12 analyzes the photo 4 through the identification algorithm 131 to distinguish the target object 31 and background information in the photo 4. In addition, the processing unit 12 executes the de-keying process through the image editing software 132 to delete the background information in the photo 4, thereby retaining the image of the target object 31 and generating an object image corresponding to the target object 31. Furthermore, the processing unit 12 performs a conversion operation on the object image to generate an AR image 5 corresponding to the target object 31.

In an embodiment, the processing unit 12 mainly converts the generated object image into a 2D AR image, but it is not limited. As shown in FIG. 1, the storage unit 13 may also store a 3D database 133 that records three-dimensional information of a plurality of target objects. In another embodiment, the electronic device 12 can also convert the generated object image into a 3D AR image according to the three-dimensional information in the 3D database 133, so as to facilitate subsequent use by the user.

FIG. 1 is an example of storing the 3D database 133 in the storage unit 13 of the electronic device 1 as an example. In other embodiments, the electronic device 1 can also be connected to the Internet through a wireless communication unit (not shown), and when the object image is to be converted into a 3D AR image, the remote 3D data can be connected through the Internet Library to access the 3D information in the 3D database without limitation.

Next, as shown in FIG. 3C, the user can edit the generated AR image 5 through the input unit 15 to further generate the edited AR image 6. Specifically, the user can use the image editing software 132 through the input unit 15 (eg, touch screen) of the electronic device 1 to perform operations such as zooming in, zooming out, rotating, stretching, deforming, layer replacement, or placement on the AR image 5 Text and other editing actions.

Next, as shown in FIG. 3D, the electronic device 1 displays the edited AR image 6 through the display unit 14. The user can operate the electronic device 1 and align the edited AR image 6 displayed on the electronic device 1 with the modification target 7 in the environment (in this embodiment, the second user is taken as an example). In this way, the electronic device 1 can directly display on the display unit 14 the appearance of the user when the target object (tie) is actually applied to the modified target (second user). In this embodiment, the user can determine whether the tie is suitable for the second user by generating and editing the AR image 6 to determine whether to purchase the tie.

For another example, the user can operate the electronic device 1 to take a picture of the watch displayed in the cabinet, and the processing unit 12 generates an AR image of the watch. The user can point the AR image of the watch to the wrist to simulate the appearance of the user after wearing the watch, thereby judging whether to purchase the watch.

For another example, the user can operate the electronic device 1 to take a photo of the tile published in the catalog, and the processing unit 12 generates an AR image of the tile. The user can aim the AR image of the ceramic tile at the position of the floor to be laid to simulate the appearance of the floor after laying the ceramic tile, thereby judging whether to purchase the ceramic tile.

For another example, the user can operate the electronic device 1 to take a picture of a curtain in a neighbor's house, and the processing unit 12 generates an AR image of the curtain. The user can aim the AR image of the curtain at the position of the window in the home to simulate the appearance of the window in the home after the curtain is installed, thereby judging whether to purchase the curtain.

In other embodiments, the electronic device 1 may also be an endoscope device in a hospital, for example. In this embodiment, the physician can use the endoscope lens to take pictures of normal parts, abnormal parts, or cancer cells in the patient's body, and generate the corresponding AR image after the back removal procedure or the anti-back removal procedure. In this way, the physician can use the AR image displayed on the display to perform the perspective positioning of the normal part, the abnormal part or the cancer cell, so that the patient can more easily understand his physical condition.

In addition, in addition to generating the above-mentioned AR image to facilitate the user to apply the AR image to modify the type on the target, the user can also edit the AR image by the processing method provided by the present invention, thereby generating an AR video.

Please refer to FIG. 4 for the first specific embodiment of the image processing flowchart of the present invention.

FIG. 4 reveals an image processing flow similar to FIG. 2. Specifically, the user can operate the electronic device 1 to align the camera lens 11 with the target object (step S30), and take a photo including at least the target object (step S32). Next, the processing unit 12 recognizes the target object from the photo through the recognition algorithm 131 (step S34), and performs a backtracking process on the photo through the image editing software 132 Sequence to generate an object image corresponding to the target object (step S36). And, the processing unit performs conversion processing on the generated object image to convert the object image into an AR image (step S38).

Then, the processing unit 12 further executes the image editing software 132, wherein the electronic device 1 can automatically or manually operate the image editing software 132 to edit the generated AR image (step S40).

In the embodiment of FIG. 4, the processing unit 12 continues to determine whether the editing action of the electronic device 1 or the user is completed after step S40 (step S42 ), and returns to step S40 before the editing action is completed to continue to pass the image editing software 132 To edit AR images.

In this embodiment, the electronic device 1 or the user can perform multiple editing operations on the AR image through the image editing software 132 to generate multiple continuous AR images, and the multiple AR images constitute an AR video. If the processing unit 12 determines in step S40 that the editing operation has been completed, the processing unit 12 may output an edited movie composed of a plurality of edited AR images (step S44). In an embodiment, the processing unit 12 may determine that the editing operation has been completed when the user triggers the save button or stops editing for a set period of time, but it is not limited.

Through the steps shown in FIG. 4, the user can use the target object of interest as creative material, capture the image of the target object through the camera lens 11 of the electronic device 1, and pass the processing unit 12, recognition algorithm 131, and image editing The software 132 and the 3D database 133 create 2D or 3D AR videos based on the image of the target object. This can effectively improve the creative quality and creative speed of AR movies.

Through the above technical solution of the present invention, the user can easily convert the objects seen and interested in life into AR images in real time, thereby benefiting the user to apply the object to the modification target or create based on the object The corresponding AR movie is quite convenient.

The above is only a preferred specific example of the present invention, and the patent scope of the present invention is not limited by this, so all equivalent changes in applying the content of the present invention are included in the scope of the present invention in the same way. Bright.

1: Electronic device

11: Camera lens

12: Processing unit

13: Storage unit

131: Identification algorithm

132: Image editing software

133: 3D database

14: display unit

15: input unit

Claims (10)

An augmented reality image processing method, applied to an electronic device, includes the following steps: a) shooting a target object published in a magazine or displayed on a computer screen through a camera lens of the electronic device to generate a photo B) The electronic device recognizes the target object in the photo through an identification algorithm; c) The electronic device performs a de-keying process on the photo through an image editing software to generate an object image corresponding to the target object; d) Converting the object image into an augmented reality image, wherein the augmented reality image is an image that can coexist with the image around the reality; d1) The camera lens of the electronic device continuously captures an instant of the surrounding environment An image; e) simultaneously displaying the real-time image and the augmented reality image on a display unit of the electronic device; and f) moving the electronic device to align the augmented reality image displayed on the display unit with the real-time image A modified target in the image, thereby simulating the appearance of the modified target after applying the target object. The method for processing augmented reality images according to claim 1, further comprising the following steps: d1) after the step d), the augmented reality images are edited by the image editing software; wherein, the step e ) Shows the augmented reality image after editing. The processing method of augmented reality image according to claim 2, wherein the step d1) is to enlarge, reduce, stretch, deform, replace the layer or place text by using the image editing software Editing actions. The processing method of augmented reality image as described in claim 2, wherein in step b), the identification algorithm is to analyze the photo by an artificial intelligence algorithm completed in advance to select from the photo The target object is identified, wherein the artificial intelligence algorithm recognizes the appearance of the multiple target object through training. The processing method of augmented reality image according to claim 4, wherein the artificial intelligence algorithm analyzes the image, color and shape characteristics of the photo to distinguish the target object and background information from the photo, And the electronic device removes the background information from the photo and retains the target object to generate the object image when performing the back-removal process. The processing method of augmented reality image as described in claim 1, further comprising the following steps: a0) before the step a, placing the target object in a target background; wherein, the photographed in the step a) The photo contains the target object and the target background. The method for processing augmented reality images according to claim 2, further comprising the following steps: g) after the step d1), determine whether the editing action is completed; h) repeatedly execute the editing action before the editing action is completed Step d1); and i) After the editing operation is completed, an edited video composed of a plurality of edited augmented reality images is output. The processing method of augmented reality image as described in claim 2, wherein the image editing software is Photoshop software. The processing method of augmented reality image according to claim 1, wherein the step d) is to convert the object image into a 2D augmented reality image. The processing method of augmented reality image according to claim 1, wherein the electronic device is connected to a 3D database, and the step d) is to convert the object image into a 3D augmentation based on the three-dimensional information in the 3D database Reality images.

Images