JP2020534590A - Processing of visual input - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for processing a visual input. An image processing method according to an embodiment provides meaningful information associated with at least one frame of a video stream continuously captured by a camera and / or a user interface associated with the meaningful information. be able to. [Selection diagram] Fig. 3

Description

The present invention relates to techniques for processing visual inputs. More specifically, image processing methods and systems that can provide meaningful information associated with at least one frame of a video stream continuously captured by a camera and / or a user interface associated with the meaningful information. The present invention relates to a computer program recorded on a computer-readable storage medium for causing the computer to execute the image processing method according to the embodiment of the present invention in combination with a computer, and the storage medium thereof.

There are a variety of prior art techniques for processing visual inputs such as images. For example, Korean Patent Application Publication No. 10-2003-0024786 analyzes the entire image taken by a digital camera in relation to text information, and recognizes and interprets the information by OCR (Optical Character Reader) technology or the like. It discloses a technique for recording as compressed text code for additional processing and / or transmission.

To provide meaningful information associated with at least one frame of a video stream continuously captured by a camera and / or a user interface associated with the meaningful information.

Image processing methods and systems that can provide meaningful information associated with at least one frame of a video stream continuously captured by a camera and / or a user interface associated with the meaningful information, combined with a computer. The present invention provides a computer program recorded on a computer-readable storage medium for causing a computer to execute the image processing method according to the embodiment of the present invention, and a storage medium thereof.

An image processing method, in which a camera module is executed in response to entering an image processing mode, and a stage in which an image stream is continuously input in the executed camera module and displayed on the screen in order. The stage of transmitting the input image stream to the recognition engine, and when there is a recognition result recognized by the recognition engine for the input image stream, the input image stream is displayed. Provided is an image processing method comprising a step of displaying the recognition results in order on the screen inside.

An image processing method in which an image captured by a camera module of an electronic device is received via a network, a step of generating a detect animation of an object included in the received image, and a step of generating the generated detect animation. An image processing method comprising a step of transmitting to the electronic device, a step of generating an image search result of the received image, and a step of transmitting the generated image search result to the electronic device. provide.

Provided is a computer program recorded on a computer-readable storage medium for combining with a computer to cause the computer to execute the image processing method.

Provided is a computer-readable storage medium in which a program for causing a computer to execute the image processing method is recorded.

The executed camera module comprises at least one processor implemented to execute a computer-readable instruction, the at least one processor causing the camera module to execute in response to entering image processing mode. In, the input of the image stream is continuously received and displayed on the screen in order, the input image stream is transmitted to the recognition engine, and the recognition recognized by the recognition engine with respect to the input image stream. Provided is a computer device characterized in that, when a result exists, the stream of the input image executes an operation of displaying the recognition result in order on a screen being displayed.

An image captured in a camera module of an electronic device is received over a network by the at least one processor, which comprises at least one processor implemented to execute a computer-readable instruction, and the received image. Generates a detect animation of the object included in the device, transmits the generated detect animation to the electronic device, generates an image search result of the received image, and transmits the generated image search result to the electronic device. Provided is a computer device characterized by performing an action to be performed.

It is possible to provide meaningful information associated with at least one frame of a video stream continuously captured by a camera and / or a user interface associated with the meaningful information.

It is a figure which showed the example of the network environment in one Embodiment of this invention. It is a block diagram for demonstrating the internal structure of an electronic device and a server in one Embodiment of this invention. It is a figure which showed the example of the execution environment of the image processing system in one Embodiment of this invention. It is a figure which showed the example which provides the additional information or the additional function by OCR recognition in one Embodiment of this invention. It is a figure which showed the example which provides the additional information or the additional function by OCR recognition in one Embodiment of this invention. It is a figure which showed the example which adjusts the font size by the number of recognized characters in one Embodiment of this invention. It is a figure which showed the example which adjusts the font size by the number of recognized characters in one Embodiment of this invention. It is a figure which showed the example which adjusts the font size by the number of recognized characters in one Embodiment of this invention. It is a figure which showed the example which provides the additional function by bar code recognition in one Embodiment of this invention. It is a figure which showed the example which provides the additional function by QR code (registered trademark) recognition in one Embodiment of this invention. It is a figure which showed the example which provides the image search result in one Embodiment of this invention. It is a figure which showed the example which provides the image search result in one Embodiment of this invention. It is a figure which showed the example which provides the image search result in one Embodiment of this invention. It is a figure which showed the example which provides the image search result in one Embodiment of this invention. It is a figure which showed the example which provides the image search result in one Embodiment of this invention. It is a figure which showed the example which provides the image search result in one Embodiment of this invention. It is a figure which showed the example which extracts the main color and limits the saturation in one Embodiment of this invention. It is a figure which showed the example which provides the image search result by the template designed by the type in one Embodiment of this invention. It is a figure which showed the example which provides the image search result by the template designed by the type in one Embodiment of this invention. It is a figure which showed the example which provides the image search result by the template designed by the type in one Embodiment of this invention. It is a figure which showed the example which displays the detect animation in one Embodiment of this invention. It is a figure which showed the example which displays the detect animation in one Embodiment of this invention. It is a figure which showed the example which displays the detect animation in one Embodiment of this invention. It is a figure which showed the example which displays the detect animation in one Embodiment of this invention. It is a figure which showed the example of the detect animation in one Embodiment of this invention. It is a flowchart which showed the example of the image processing method in one Embodiment of this invention. It is a flowchart which showed the other example of the image processing method in one Embodiment of this invention. It is a flowchart which showed still another example of the image processing method in one Embodiment of this invention. It is a flowchart which showed the example of the method of generating the detect animation in one Embodiment of this invention. It is a figure which showed the example which provides additional information by place recognition in one Embodiment of this invention. It is a figure which showed the example which provides additional information by recognition of an image code in one Embodiment of this invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

The image processing method according to an embodiment of the present invention may be performed by a computer device such as an electronic device and / or a server described below. At this time, in the computer device, the computer program according to the embodiment of the present invention may be installed and executed, and the computer device may perform image processing according to the embodiment of the present invention under the control of the executed computer program. You may carry out the method. The computer program described above may be recorded on a computer-readable storage medium in combination with a computer device to allow the computer to perform an image processing method.

FIG. 1 is a diagram showing an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. Such FIG. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited as in FIG.

The plurality of electronic devices 110, 120, 130, 140 may be fixed terminals or mobile terminals realized by a computer device. Examples of a plurality of electronic devices 110, 120, 130, 140 include smartphones, mobile phones, navigation systems, PCs (personal computers), notebook PCs, digital broadcasting terminals, PDAs (Personal Digital Assistants), and PMPs (Tablet Multimedia Players). ), Tablets, etc. As an example, FIG. 1 shows a smartphone as an example of the electronic device 1 (110), but in the embodiment of the present invention, the electronic device 1 (110) substantially uses a wireless or wired communication method. It may mean one of a variety of physical computer devices capable of communicating with other electronic devices 120, 130, 140 and / or servers 150, 160 via network 170.

The communication method is not limited, and not only the communication method using the communication network (for example, mobile communication network, wired Internet, wireless Internet, broadcasting network) that can be included in the network 170, but also the short distance between devices. Wireless communication may be included. For example, the network 170 includes a PAN (personal area network), a LAN (local area network), a CAN (campus area network), a MAN (metropolitan area network), a WAN (wide network) network, and the like. It may include any one or more of the networks. In addition, network 170 may include any one or more of network topologies including bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks, and the like. There is no limitation.

Each of the servers 150, 160 is realized by one or more computer devices that communicate with a plurality of electronic devices 110, 120, 130, 140 via the network 170 to provide instructions, codes, files, contents, services, and the like. You can. For example, the server 150 may be a system that provides the first service to a plurality of electronic devices 110, 120, 130, 140 connected via the network 170, and the server 160 may also be a plurality of systems connected via the network 170. It may be a system that provides a second service to electronic devices 110, 120, 130, 140. As a more specific example, the server 150 is a service (for example, the present implementation) aimed at by the application through an application which is a computer program installed and executed in a plurality of electronic devices 110, 120, 130, 140. In addition to the service for image processing related to the form, an information providing service, a messaging service, a mail service, a content transmission service, etc.) may be provided to a plurality of electronic devices 110, 120, 130, 140 as the first service. .. As another example, the server 160 may provide a service as a second service that distributes files for installing and executing the above-mentioned application to a plurality of electronic devices 110, 120, 130, 140.

FIG. 2 is a block diagram for explaining the internal configurations of the electronic device and the server according to the embodiment of the present invention. FIG. 2 describes the internal configuration of the electronic device 1 (110) and the internal configuration of the server 150 as examples for the electronic device. Further, the other electronic devices 120, 130, 140 and the server 160 may have the same or similar internal configuration as the electronic device 1 (110) or the server 150 described above.

Electronic device 1 (110) and server 150 may include memory 211,221, processors 212,222, communication modules 213 and 223, and input / output interfaces 214 and 224. The memories 211 and 221 are computer-readable storage media and may include a permanent mass storage device such as a RAM (random access memory), a ROM (read only memory), and a disk drive. Here, a permanent large-capacity storage device such as a ROM or a disk drive may be included in the electronic device 1 (110) or the server 150 as another permanent storage device that is separated from the memories 211 and 221. In addition, the memories 211 and 221 contain an operating system and at least one program code (for example, an electronic device 1 (110) for providing a browser installed and executed in the electronic device 1 (110) or a specific service. The code for applications etc. installed in) may be recorded. Such software components may be loaded from a computer-readable storage medium separate from the memories 211 and 221. Such other computer-readable storage media may include computer-readable storage media such as floppy® drives, disks, tapes, DVD / CD-ROM drives, memory cards, and the like. In other embodiments, software components may be loaded into memory 211,221 through communication modules 213, 223, which are not computer readable storage media. For example, at least one program is a computer program installed by a file provided via network 170 by a file distribution system (eg, server 160 described above) that distributes developer or application installation files (eg, described above). It may be loaded into the memory 211 or 221 based on the application.

Processors 212 and 222 may be configured to process instructions in a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to processors 212 and 222 by memory 211, 221 or communication modules 213 and 223. For example, processors 212 and 222 may be configured to execute instructions received according to program code recorded in storage devices such as memories 211 and 221.

The communication modules 213 and 223 may provide a function for the electronic device 1 (110) and the server 150 to communicate with each other via the network 170, and the electronic device 1 (110) and / or the server 150 may provide other functions. A function for communicating with an electronic device (for example, electronic device 2 (120)) or another server (for example, server 160) may be provided. As an example, a request generated by the processor 212 of the electronic device 1 (110) according to a program code recorded in a storage device such as a memory 211 is transmitted to the server 150 via the network 170 under the control of the communication module 213. You can. On the contrary, control signals, instructions, contents, files, etc. provided under the control of the processor 222 of the server 150 pass through the communication module 223 and the network 170, and then through the communication module 213 of the electronic device 1 (110). It may be received at 1 (110). For example, control signals, commands, contents, files, etc. of the server 150 received through the communication module 213 may be transmitted to the processor 212 and the memory 211, and the contents, files, etc. are further included in the electronic device 1 (110). It may be recorded on a storage medium (permanent storage device described above).

The input / output interface 214 may be a means for an interface with the input / output device 215. For example, the input device may include a device such as a keyboard or a mouse, and the output device may include a device such as a display or a speaker. As another example, the input / output interface 214 may be a means for an interface with a device such as a touch screen in which functions for input and output are integrated into one. The input / output device 215 may be composed of an electronic device 1 (110) and one device. Also, the input / output interface 224 of the server 150 may be a means for connecting to the server 150 or for interfacing with a device (not shown) for input or output that the server 150 can include. As a more specific example, when the processor 212 of the electronic device 1 (110) processes the instruction of the computer program loaded in the memory 211, the processor 212 is configured by using the data provided by the server 150 and the electronic device 2 (120). The service screen or content to be generated may be displayed on the display through the input / output interface 214.

Also, in other embodiments, the electronic device 1 (110) and the server 150 may include more components than the components of FIG. However, most prior art components need not be clearly illustrated. For example, the electronic device 1 (110) may be realized to include at least a part of the above-mentioned input / output device 215, a transceiver, a GPS (Global Positioning System) module, a camera, various sensors, and a database. It may further include other components such as. As a more specific example, when the electronic device 1 (110) is a smartphone, the acceleration sensor, gyro sensor, camera module, various physical buttons, buttons using the touch panel, input / input, which are generally included in the smartphone. Various components such as an output port, a vibrator for vibration, and the like may be realized to be further included in the electronic device 1 (110).

FIG. 3 is a diagram showing an example of an execution environment of an image processing system according to an embodiment of the present invention. In FIG. 3, electronic device 1 (110) includes a camera module 310, an OCR recognition engine 320, a barcode recognition engine 330, and a QR code recognition engine 340, and a server 150 includes an image search engine 350 and a detect animation generation engine 360. An example including is shown. Depending on the embodiment, the detect animation generation engine 360 may be included in the electronic device 1 (110).

Each engine 320-360 may be implemented in the form of a software module. For example, the OCR recognition engine 320, the barcode recognition engine 330, and the QR code recognition engine 340 included in the electronic device 1 (110) are functional expressions provided by an application installed and executed in the electronic device 1 (110). It may be there. In this case, the processor 212 of the electronic device 1 (110) may execute the operation by the OCR recognition engine 320, the barcode recognition engine 330, and the QR code recognition engine 340 according to the code of the application. Similarly, the image search engine 350 and the detect animation generation engine 360 included in the server 150 may be realized in the form of software modules, and may be functional expressions provided by a computer program executed on the server 150. In this case, the processor 222 of the server 150 may execute the operation by the image search engine 350 and the detect animation generation engine 360 according to the code of the computer program.

The OCR recognition engine 320 may recognize one or more letters and / or numbers from within the image.

The barcode recognition engine 330 may recognize the barcode from within the image.

The QR code recognition engine 340 may recognize the QR code from within the image.

The image search engine 350 may receive input of an image and return various search results (image, text, etc.) related to the image.

The detect animation generation engine 360 may generate and provide a detect animation for visually expressing the process of searching for an object in an image. Such a detect animation may be utilized as an effect for inducing the user's interest in the waiting time until the search result is obtained and giving the impression that the display of the search result is not slow.

When the electronic device 1 (110) enters the image processing mode according to the present embodiment, the electronic device 1 (110) may execute the camera module 310, and the camera module may be executed without any special input from the user. The image stream continuously captured by the 310 may be transmitted as inputs to the OCR recognition engine 320, the barcode recognition engine 330, and the QR code recognition engine 340. At this time, each of the OCR recognition engine 320, the barcode recognition engine 330, and the QR code recognition engine 340 analyzes the image (frame) of the input image stream in order, and each corresponds object (character, number, bar code, The QR code, etc.) may be recognized from within the image.

As mentioned above, the OCR recognition engine 320 may attempt to recognize the characters and / or numbers contained in the image by sequentially analyzing the images contained in the image stream and return the recognized characters and / or numbers. You can do it. In this case, electronic device 1 (110) may display the returned characters and / or numbers on the screen and may provide additional information and functions related to the displayed characters and / or numbers. For example, when a character in the first language is recognized, the electronic device 1 (110) is a user for accessing a translation function capable of translating the recognized character in the first language into a character in another language. An interface may be provided. As another example, electronic device 1 (110) may provide a user interface for accessing a search function that utilizes the returned letters and / or numbers as keywords. As yet another example, the electronic device 1 (110) may automatically perform a search using the returned characters and / or numbers as keywords and provide the search results.

The barcode recognition engine 330 may attempt to recognize the barcode contained in the image by sequentially analyzing the images contained in the image stream, and may return information on the recognized barcode. In this case, electronic device 1 (110) may provide additional information or additional functions associated with the returned information. For example, the electronic device 1 (110) searches for information corresponding to the corresponding bar code (for example, information on books, wine, etc. corresponding to the bar code) based on the information on the returned bar code, and the search is performed. The results may be provided as additional information. As another example, electronic device 1 (110) may provide a user interface for accessing information corresponding to a barcode.

The QR code recognition engine 340 may try to recognize the QR code contained in the image by sequentially analyzing the images included in the image stream, and may return the information regarding the recognized QR code. In this case, the electronic device 1 (110) may provide additional information and additional functions related to the returned information. As in the case of the barcode, the electronic device 1 (110) may provide the information corresponding to the recognized QR code (for example, the information corresponding to the URL included in the QR code) as additional information. Alternatively, a user interface for accessing information corresponding to the recognized QR code may be provided.

As described above, when the electronic device 1 (110) enters the image processing mode, the electronic device 1 (110) executes the camera module 310 without any special input from the user and is provided in the camera module 310. Automatically search for objects such as letters, numbers, bar codes, QR codes, etc. from each image in the image stream, and automatically provide additional information and functions related to the searched object. It becomes possible to do.

On the other hand, the electronic device 1 (110) is a preset user such as a user selecting a shooting button (for example, the user touches a shooting button area displayed on the touch screen in a touch screen environment with a finger). The occurrence of user input may be monitored at the interface. In FIG. 3, process 370 monitors the occurrence of user input, and when user input occurs, shows an example in which the captured image is transmitted to the server 150 according to the user input. As a more specific example, as described with reference to FIG. 2, the electronic device 1 (110) may transmit the image captured via the network 170 to the server 150 by using the communication module 213. The server 150 may receive the captured image transmitted via the network 170 by using the communication module 223.

The server 150 may provide the transmitted image to the image search engine 350 and the detect animation generation engine 360, respectively.

As described above, the image search engine 350 may receive the image captured and transmitted by the electronic device 1 (110) as input and return various search results related to the image. For example, the image search engine 350 may recognize an object included in the image, search for an image, a document, a text, or the like related to the recognized object, and return the object. As a more specific example, if a puppy included in an image is recognized and this puppy type is analyzed as a "retriever", search results such as images and documents related to the "retriever" will be generated and returned. You can. The server 150 may transmit the returned search result to the electronic device 1 (110) via the network 170, and the electronic device 1 (110) may provide the corresponding search result to the user. Depending on the embodiment, the electronic device 1 (110) adds not only the image but also the time when the image is captured, the current position of the electronic device 1 (110), the user information of the electronic device 1 (110), and the like to the server 150. You may send it with. In this case, the server 150 may further provide search results based on at least one of location, time, and user information. For example, among various search results related to an image, a search result related to the user's current position or a search result related to time may acquire a priority in displaying the search result.

The detect animation generation engine 360 receives an image captured and transmitted by the electronic device 1 (110) as an input, and generates a detect animation for visually expressing the process of searching for an object in the corresponding image. Good. In this case as well, the server 150 may transmit the generated detect animation to the electronic device 1 (110) via the network 170, and the detect animation is applied to the electronic device 1 (110) over a preset time. By displaying in association with the image of, the user of the electronic device 1 (110) obtains a search result (a search result returned by the image search engine 350 described above and provided to the electronic device 1 (110) by the server 150). It may be used as an effect for inducing the interest of the corresponding user in the waiting time until the search result is displayed so as not to be slow. Such a detect animation may basically consist of a plurality of points at positions related to the object to be searched from the image and a line connecting such points, and the expression of the points and the points are connected. The representation of the line to be done may be displayed as an animation effect. In addition, depending on the embodiment, the thickness, size, brightness, color, and the like of points and lines may be changed to give an additional animation effect. Further, the surfaces composed of the points connecting the points may be displayed in different colors to give a three-dimensional effect, or an embodiment in which the lines connecting the points are curved may be considered. Such detect animation will be described in more detail below.

4 and 5 are diagrams showing an example of providing additional information or additional functions by OCR recognition in one embodiment of the present invention.

FIG. 4 shows screen examples 410 to 440 of the electronic device 1 (110). In the first screen example 410, as described with reference to FIG. 3, the image stream captured by the camera module 310 is automatically transmitted to the OCR recognition engine 320, and the characters are recognized in real time by the OCR recognition engine 320. An example is shown.

At this time, the second screen example 420 shows an example in which the color of the recognized character is changed and displayed on the image as the recognition is completed, and the T (TEXT) character recognition button is displayed.

Further, the third screen example 430 shows an example in which functions related to the recognized character, such as copy, translation, and reading for the recognized character, are provided with the selection of the T character recognition button. .. For example, the copy function may be a function for copying the recognized character to the clipboard. Further, the translation function may be a function for translating the recognized characters of the first language into the characters of the second language. Further, the reading function may be a function of reading out the recognized first character, and may be a function of generating and outputting audio corresponding to the first character.

The fourth screen example 440 shows an example in which the recognized characters in the first language are translated into the characters in the second language and displayed as the translation function is selected by the user. At this time, the function for viewing the details displayed in the fourth screen example 440 includes a search result of the language dictionary for the recognized first language character, a search result using the recognized first language character as a keyword, and the like. It may be a function for providing additional content or moving to a separate translation result page. In addition, if the recognition is not accurate or the translation result is not satisfactory, a handwriting search function is further provided to directly write characters using the handwriting recognition function button and execute the search. You may.

In the embodiment of FIG. 4, an example of providing an additional function for the entire recognized character has been described, but depending on the embodiment, a part of the recognized characters is selected and a part of the selected characters is selected. Applicable additional functions may be provided for.

FIG. 5 shows screen examples 510 to 530 of the electronic device 1 (110). At this time, the first screen example 510 shows an example in which the image is displayed on the screen before the text (characters and / or numbers) is recognized.

Further, the second screen example 520 shows an example in which the recognized text is displayed at a position similar to the position where the corresponding text is displayed in the image as the text is recognized.

Further, the third screen example 530 shows an example in which the recognized text is reconstructed and displayed so that the user can easily see it. At this time, the third screen example 530 shows an example in which the image is darkened so that the recognized text is expressed more appropriately. Further, the third screen example 530 describes an example in which additional functions such as a copy function, a translation function, and a reading function are provided in connection with the recognized text.

At this time, the text displayed in the second screen example 520 and the third screen example 530 has an animation effect such as a color change, a change in the frame in which the text is displayed from two dimensions to three dimensions, and the like. It may be displayed with an animation effect.

6 to 8 are diagrams showing an example of adjusting the font size according to the number of recognized characters in one embodiment of the present invention.

FIG. 6 shows an example of displaying the recognized text in 80 px size when the number of recognized characters is 1 to 6.

Further, FIG. 7 shows an example in which the recognized text is displayed in a size of 60 px when the number of recognized characters is 7 to 40 characters.

Further, FIG. 8 shows an example in which the recognized text is displayed in a size of 40 px when the number of recognized characters is 41 or more.

As described above, when displaying the recognized text, the electronic device 1 (110) can automatically adjust and display the font size according to the number of characters of the text recognized from the image.

FIG. 9 is a diagram showing an example of providing an additional function by bar code recognition in one embodiment of the present invention.

FIG. 9 shows screen examples 910 and 920 of electronic device 1 (110). The first screen example 910 shows an example in which an image including a barcode is displayed, and the second screen example 920 displays a barcode button as the barcode recognition engine 330 recognizes the barcode. An example is shown. When the user selects the displayed barcode button, product information corresponding to the barcode (for example, book information, wine information, etc.) may be provided.

FIG. 10 is a diagram showing an example of providing an additional function by QR code recognition in one embodiment of the present invention.

FIG. 10 shows screen examples 1010 and 1020 of electronic device 1 (110). The first screen example 1010 shows an example in which an image including a QR code is displayed, and the second screen example 1020 displays a QR code button as the QR code is recognized by the QR code recognition engine 340. An example is shown. When the user selects the displayed QR code button, the page of the URL including the corresponding QR code may be landed.

11 to 15 are diagrams showing an example of providing an image search result in one embodiment of the present invention.

FIG. 11 shows screen examples 111 to 1130 of the electronic device 1 (110). The first screen example 1110 shows an example in which an image is displayed, and the second screen example 1120 shows an example in which an object is searched from within the image. At this time, the image may be transmitted to the server 150 when the user presses the shooting button, and the server 150 may perform the image search and the detect animation generation in the image search engine 350 and the detect animation generation engine 360. .. In the second screen example 1120, a detect animation composed of a plurality of points and a line connecting the points is provided from the server 150 and displayed on the screen to search for the face of a dog (dog). Is shown as an example of visually informing the user. Further, the third screen example 1130 shows an example in which "puppy" and "Labrador retriever" are displayed as text information (related keywords) related to the searched object, and the image search result of the image is further displayed. There is. Such text information and image search results may be provided by the server 150. At this time, if the user selects the area where the "puppy" or "Labrador retriever" displayed as the text information related to the searched object is displayed, the text search result using the corresponding text information as a keyword. May be provided to the user. Further, when each of the image search results is selected by the user, the page corresponding to the corresponding search result may be landed.

FIG. 12 shows screen examples 121 to 1240 of electronic device 1 (110). The first screen example 1210 shows an example in which an image is displayed, and the second screen example 1220 and the third screen example 1230 display a detect animation composed of a plurality of points and a line connecting the points. By doing so, an example of the process of visually informing the user that the fish is being searched for is shown. In addition, the third screen example 1240 shows an example in which "ornamental fish" and "Asia arowana" are displayed as text information (related keywords) related to the searched object, and the image search result of the image is further displayed. ing. In this case as well, when the area displaying "Aquarium fish" or "Asian arowana" displayed as text information related to the searched object is selected by the user, the corresponding text information is used as a keyword. Text search results may be provided to the user. Further, when each of the image search results is selected by the user, the page corresponding to the corresponding search result may be landed.

13 to 15 also show an example of the process of providing image search results, as in FIGS. 11 and 12. FIG. 13 shows the process of detecting a flower along the outer line to recognize an object and providing a detect animation and an image search result, and FIG. 14 shows a process of detecting a cat's face along the outer line to recognize an object. The process of providing the detect animation and the image search result is shown, and FIG. 15 shows the process of detecting the Eiffel tower along the outer line to recognize the object and providing the detect animation and the image search result, respectively. ..

FIG. 16 is a diagram showing an example of providing an image search result in one embodiment of the present invention, and FIG. 17 is an example of extracting a main color and limiting saturation in one embodiment of the present invention. It is a figure which showed. FIG. 16 shows an example in which the main colors of an image are extracted by the autotopic technique and the extracted main colors are used for displaying related keywords and image search results. At this time, in consideration of the visibility of the text, the Saturation (S) and the brightness (B) values may be limited to the range of 50 to 70% in the HSB (Hue-Saturation-Brightness) value. FIG. 17 shows an example in which the main colors are extracted by a color picker and the saturation value is limited to 50%.

18 to 20 are diagrams showing an example of providing an image search result in one embodiment of the present invention. Figures 18 to 20 show "person_domestic", "person_group", "person_overseas", "encyclopedia_animal", "encyclopedia_masterpiece", "encyclopedia_wine", "famous place", "region", As a template pre-designed by various types such as "domestic_place", an example of providing a specific search result among image search results in the form of a correct answer type card is shown. Also in this case, the saturation and lightness values may be limited to the range of 50 to 70% by the HSB (Hue-Saturation-Brightness) value in consideration of the visibility of the text.

21 to 24 are diagrams showing an example of displaying a detect animation in one embodiment of the present invention.

The method of generating the detect animation is as follows. For example, the detect animation generation method may be executed by the detect animation generation engine 360 described above.

(1) Preprocessing process: A process of executing preprocessing required for image search, such as grayscale, blow, and edge detection.

(2) Object Detection: A process of searching for an object in the input video and generating a bounding box containing the object. A generally well-known object search technique may be used to search for an object, and the following process may be performed on the generated bounding box area.

(3) The process of extracting meaningful feature points from the outline of an object. As an example, a preset number (for example, about several hundred) of feature points may be extracted using the FAST algorithm.

(4) A process of generating a convex hull of the extracted feature points. As an example, a block polygon containing all the feature points in question may be generated as a convex package, and the generation of such a convex package may be generated using a well-known algorithm.

(5) When the number of points constituting the convex hull does not reach a predetermined number (6 as an example), the process of adding points until the required number is reached while performing the addition step. As an example, the closest feature point may be selected from the middle of the two points forming the convex hull and used as a point for forming the convex hull.

(6) A process of constructing a plurality of points (outer points) forming an outer line with a plurality of points selected in (4) and (5) described above and calculating the center coordinates (center points) of the outer points. As an example, the calculation may be performed with the point (average of X value, average of Y value) of each coordinate as the center point.

(7) For each outer point, the process of selecting the feature point closest to the intermediate value between the outer point and the center point.

(8) The process of repeating steps (6) and (7) when trying to add a line inside. It may be omitted when connecting only one-step lines.

(9) The process of returning the coordinates of points and the order of animation, or the generated animation.

The process of generating the detect animation by the processes (1) to (9) may be transformed into various forms depending on the design as one embodiment.

For example, FIG. 21 shows a detect animation of a triangular structure in which a convex package composed of five points and a point of depth 3 are connected. More specifically, in FIG. 21, a pentagonal dot is formed and connected to the contour of an object which is a convex hull, a debs 2 dot is formed and connected to the pentagon dot, and the left and right sides of the debs 2 dot are connected to form a debs 3 dot. An example is shown in which a form of detect animation is generated by forming a central point, which is a central point, and connecting two dots of Debs and three dots of Debs vertically. At this time, an example is shown in which the Debs 2 dots are formed at 1/3 of the line connecting the center and the center point (Debs 3 dots) of the connecting line of the pentagonal dots.

Further, FIG. 22 shows a detect animation of a triangular and quadrangular structure in which a convex hull composed of nine points and the points of Debs 3 are connected. In FIG. 22 such as this, the coordinates (for example, the start of the contour dots) in which the color difference is equal to or larger than the preset difference on the straight line from the contour dots (nine points constituting the convex hull) to the center dot (center point) Dots are displayed at the points where there is a difference of 20 or more in the B value among the RGB values), the dots of the same depth are connected to the left and right, and the dots of the closest distance are connected between the dots of the upper and lower fats. By doing so, an example of generating the form of the detect animation is shown.

Further, FIGS. 23 and 24 show an example in which a three-dimensional effect is given by different colors of the triangular structure and the transparency is adjusted at the same time. In other words, FIG. 24 shows an example in which the transparency in the triangular structure is higher than that in FIG. 23.

FIG. 25 is a diagram showing an example of detect animation in one embodiment of the present invention. In FIG. 25, the detect animation for the Eiffel Tower described with reference to FIG. 15 is not merely an image form, but repeats the display of a plurality of points and a line connecting the points as shown in the pictures in FIG. 25 in order. This shows that such a detect animation is used as an effect to induce the user's interest before the search result is obtained and to give the impression that the display of the search result is not delayed. .. At this time, the detect animation may give an additional animation effect by changing the thickness, size, brightness, color, etc. of the points and lines as described above, and has been described with reference to FIGS. 23 and 24. As described above, different colors may be applied to the triangular structure or the quadrangular structure to give a three-dimensional effect, or the transparency of the colors may be adjusted to determine the degree to which the image is displayed.

FIG. 26 is a flowchart showing an example of an image processing method according to an embodiment of the present invention. The image processing method according to the present embodiment may be executed by a computer device such as the electronic device 1 (110) described above. For example, the processor 212 of the electronic device 1 (110) may be implemented to execute a control instruction (instruction) by the code of the operating system included in the memory 211 and the code of at least one computer program. Here, the processor 212 causes the electronic device 1 (110) to execute the steps 261 to 2640 included in the image processing method of FIG. 26 according to the control instruction provided by the code recorded in the electronic device 1 (110). Electronic device 1 (110) may be controlled.

At step 2610, the computer device may execute the camera module in response to entering the image processing mode. As an example, an application installed on a computer device may provide the user with a user interface for entering image processing mode. When an input to the user's user interface occurs, the computer device may execute the camera module in the image processing mode entered according to the generated input.

At step 2620, the computer device may continuously receive input of a stream of images in the camera module to be executed and display it on the screen in order. For example, when the camera is executed on the smartphone, the image input to the camera and displayed in real time on the screen of the smartphone may correspond to such a stream of images.

At step 2630, the computer device may transmit a stream of input images to the recognition engine. The camera module may continuously transmit a stream of images input in real time to the recognition engine, which may analyze the stream of images to produce the recognition result intended by the recognition engine. For example, in the embodiment of FIG. 3, the electronic device 1 (110) includes a text recognition engine such as the OCR recognition engine 320, a barcode recognition engine 330, and an image code recognition engine such as the QR code recognition engine 340. explained.

At step 2640, if there is a recognition result recognized by the recognition engine for the input image stream, the computer device further displays the input image stream as the recognition result on the screen being displayed. You can. For example, the recognition engine may include a text recognition engine that recognizes the text contained in the image input as a recognition result.

At this time, if there is a recognition result recognized by the text recognition engine, in step 2640, the computer device dynamically changes the display position of the recognized text based on the position of the text area recognized from the input image. May be adjusted to. For example, the first screen example 410 of FIG. 4 shows an example in which the recognized text is displayed at the position of the region including the text in the image. As another example, the second screen example 520 of FIG. 5 also describes an example in which the recognized text is displayed at a position similar to the position in the image of the corresponding text as the text is recognized. For example, the position of the same text area may be continuously changed in the input image by vibrating a computer device such as a smartphone by the user's hand. In this case, the computer device tracks the position of the text area by changing the position of the text area and dynamically adjusts the display position of the text so that the recognized text is displayed at the tracked position. You can.

Further, at step 2640, the computer device sequentially displays the recognized text on the screen displaying the stream of the input image, and the text area in the image in which the text is recognized among the input images is displayed. The display position of the recognized text may be determined based on the position. For example, the computer device dynamically adjusts the display position of the recognized text by changing the position of the text area as described above, and finally the text when the recognition of the text is finally completed. The recognized text may be displayed at the recognized position. In this case, the position of the text area may be persistently changed in the input image, or the display position of the recognized text is fixed even when another image without text is continuously input. May be done.

Further, the computer device may display the user interface for user confirmation for the text recognized for the input image stream in order on the screen displaying the input image stream. For example, after the text recognition engine has finally completed the recognition of the text, the computer device may handle the process of receiving user confirmation for the finally recognized text. For example, the second screen 420 of FIG. 4 shows an example of displaying a user interface for receiving confirmation of the recognized text from the user, such as a character recognition button. At this time, the stream of the image may still be displayed in real time on the screen of the computer device.

At this time, when the user confirmation occurs in the user interface for receiving the confirmation of the recognized text from the user, the computer device may display the image in which the text is displayed among the input images on the screen. For example, when the computer device recognizes the text confirmed by the user in the first image of the input images, instead of displaying the stream of the image on the screen, the computer device displays the already displayed first image on the screen. It may be displayed on. The computer device may further display at least one of a user interface for copying the recognized text and a user interface for translating the recognized text on the image displayed on the screen. .. For example, FIGS. 4 and 5 show examples of displaying a user interface for copying and translating recognized text, such as the "Copy" and "Translate" buttons.

Further, the computer device may dynamically change the size of the recognized text according to the number of characters of the recognized text and display it on the screen. For example, FIGS. 6 to 8 have described an example of displaying on the screen while dynamically reducing the size of the recognized text as the number of characters in the recognized text increases.

Further, as another example, the recognition engine may include an image code recognition engine for recognizing the image code included in the input image. In this case, in step 2640, the computer device further displays a stream of images input as a result of recognizing a link to a page corresponding to the image code recognized in the images displayed in order on the screen on the displayed screen. It may be displayed in order. For example, FIG. 9 shows an example in which a link to a page corresponding to a recognized barcode is displayed as a "bar code" button, and FIG. 10 shows a link to a page corresponding to the recognized QR code. An example of displaying as a "QR code" button is shown.

Depending on the embodiment, the recognition engine may include a plurality of recognition engines together with a text recognition engine and an image code recognition engine, in which case a stream of images input in the camera module may be input to each of the plurality of recognition engines. ..

In steps 261 to 2640 described above, the computer device automatically recognizes text, image code, and the like based on the stream of images input to the camera and displays related information before the user is photographed by the camera. Embodiments are described. In other words, in response to entering image processing mode, text and image code recognition by analyzing the entire stream of input images without the user having to press the capture button to select an image at a particular point in time. Results can be provided automatically.

On the other hand, when the user presses the shooting button and a specific image is selected, the computer device may provide the user with a function different from the above-described embodiment.

FIG. 27 is a diagram showing another example of the image processing method in one embodiment of the present invention. The image processing method according to the present embodiment may also be executed by a computer device such as the electronic device 1 (110) described above. At this time, steps 2710 to 2750 of FIG. 27 may be executed when a shooting input is generated after step 2620 of FIG. 26, and when the step of FIG. 27 is executed, steps 2630 and 2640 are omitted. May be good.

At step 2710, if a capture input occurs while sequentially displaying a stream of input images, the computer device may capture an image associated with the time of occurrence of the capture input and display it on the screen. Such a step 2710 may mean a process of taking a picture by generating a shooting input by the user.

At step 2720, the computer device may send the captured image to the server. As an example, the server may correspond to a computer device such as the server 150 described above, and the captured image may be transmitted to the server via the network 170.

At step 2730, the computer device may receive a detect animation of the object contained in the transmitted image from the server. As an example, the detect animation displays a plurality of feature points extracted from the outline of an object at positions on an image in which the plurality of feature points are extracted, and at least a part of the displayed feature points. It may include an animation that connects the feature points with a line.

At step 2740, the computer device may display the detect animation on the screen in association with the object. An example of displaying the detect animation on the screen in association with the object is as described with reference to FIGS. 11 to 15 and 21 to 25.

At step 2750, the computer device may receive the image analysis result of the transmitted image from the server and display it in association with the image displayed on the screen. The image analysis result may include the type and / or name of the object contained in the image, and may further include at least one of the images, documents, texts searched on the server in connection with the object.

FIG. 28 is a diagram showing still another example of the image processing method according to the embodiment of the present invention. The image processing method according to this embodiment may be executed by a computer device such as the server 150 described above. For example, the processor 222 of the server 150 may be implemented to execute a control instruction (instruction) by the code of the operating system included in the memory 221 and the code of at least one computer program. Here, the processor 222 may control the server 150 so that the server 150 executes the steps 281 to 2850 included in the image processing method of FIG. 28 according to the control instruction provided by the code recorded in the server 150.

At step 2810, the computer device may receive the image captured in the camera module of the electronic device over the network. Here, the electronic device may correspond to the electronic device 1 (110) described above, and the received image may correspond to the image transmitted in the step 2720 of FIG. 27. In other words, when the electronic device 1 (110) transmits the image captured by the generation of the user's shooting input via the network 170, the server 150 may receive the image in step 2810.

At step 2820, the computer device may generate a detect animation of the object contained in the received image. As described above, the detect animation displays a plurality of feature points extracted from the outline of the object at positions on the image in which the plurality of feature points are extracted, and at least one of the displayed feature points. It may include an animation that connects the feature points of the part with a line.

At step 2830, the computer device may transmit the generated detect animation to the electronic device. In this case, as described with reference to step 2730 of FIG. 27, the electronic device may receive the detect animation and, as in step 2740, display the received detect animation in association with the object on the screen.

At step 2840, the computer device may generate image search results for the received image. Image search results may include the type and / or name of the object contained in the image, and may further include images, documents, and / or text searched in association with the object.

At step 2850, the computer device may transmit the generated image search results to the electronic device. At this time, the type and / or name of the object may be displayed on the screen of the electronic device so as to overlap with the image input to the camera of the electronic device. Also, the retrieved image, document, and / or text may include a link to the corresponding page and be further displayed on the screen of the electronic device in association with the image input to the camera module of the electronic device. For example, FIG. 12 displays the types and names of recognized objects such as "ornamental fish" and "Asian arowana", and images, documents, and / or text as an additional result of Internet search by image. Is shown as an example of displaying in association with an image.

In addition, the computer device realizes the information searched as a template pre-designed for each type according to the type of information searched in relation to the object included in the received image in the form of a card and provides it to the electronic device. May be good. For example, FIGS. 18 to 20 show in advance for each type of information (person_domestic, person_group, person_overseas, encyclopedia_animal, encyclopedia_masterpiece, encyclopedia_wine, famous place, region, domestic_place, etc.). Explains an example in which a designed template exists, and the information searched as a template of the corresponding type is realized and provided in the form of a card according to the type of the searched information.

At this time, when the type and / or name of the object displayed on the electronic device is selected (for example, in the touch screen environment, the user touches the area where the type and / or name of the object is displayed with a finger). With the selection, the signal generated by the electronic device may be transmitted to the server via the network. In this case, the computer device may receive the corresponding signal and may generate a text search result using the type or name of the object as a keyword in response to the reception of the signal. The computer device may also provide the generated text search results to the electronic device. In other words, the user of the electronic device can provide the text search result of the text obtained from the image in order in addition to the image search result of the image.

FIG. 29 is a flowchart showing an example of a method of generating a detect animation according to an embodiment of the present invention. Steps 291 to 2950 included in the method of this embodiment may be included and performed in step 2820 of FIG.

At step 2910, the computer device may search for the objects contained in the received image. For example, a computer device performs preprocessing required for image search such as grayscale, blow, edge detection, etc. on a received image, then searches the image for an object, and bounds the image to include the object. A box (bounding box) may be generated. A generally well-known object search technique may be utilized for searching for such an object.

At step 2920, the computer device may extract a plurality of feature points from the contours of the object. As an example, a preset number (for example, about several hundred) of feature points may be extracted using the FAST (Fairure from Accelerated Segment Test) algorithm.

At step 2930, the computer device may generate a convex hull of the extracted feature points. As an example, a convex hull may be generated from feature points extracted using the convex hull algorithm. If the number of points for forming the convex hull does not reach the predefined number, additional feature points may be extracted.

At step 2940, the computer device may calculate the center points of the extracted feature points based on the center coordinates of a predetermined number of outer points that make up the convex hull. For example, the calculation may be performed with the point (the average of the X coordinate values and the average of the Y coordinate values) as the center with respect to the coordinates of the outer points constituting the convex package.

At step 2950, the computer device may select the feature points of the object that are closest to the center value between each of the outline points and the center point. An internal line may be added by reusing the selected feature point as the outer outline point to obtain the center point and reselecting the feature point between the outer outline point and the center point. For example, FIG. 21 shows an example in which a triangular structure detect animation is formed by connecting a convex package composed of five points and a point of Debs 3.

The generated detect animation may be transmitted to the electronic device as in step 2830 of FIG. 28, and may be associated with the object in the electronic device and displayed on the screen of the electronic device as in step 2740 of FIG. 27. At this time, the computer device may transmit information on the coordinates of the selection point including the outer point, the center point, and the closest feature point and information on the order of connecting the selection points with a line to the electronic device as a detect animation. In this case, the electronic device may display a line based on the information on the coordinates of the selected points and display an animation on the screen connecting the selected points with the line based on the information on the order. Further, the computer device may transmit the animation itself of connecting the selected points in the above order to the electronic device as a detect animation. In this case, the electronic device may display the detect animation by playing the corresponding animation in association with the object.

In the process of providing the image processing result to the user, such a detect animation indicates that the object of the image requested by the user is being analyzed, so that the waiting time until the image search result is obtained is increased. It can induce the user's interest and give the impression that the display of search results is not delayed.

FIG. 30 is a diagram showing an example in which additional information is provided by place recognition in one embodiment of the present invention. FIG. 30 shows screen examples 3010 and 3020 of electronic device 1 (110). In the first screen example 3010, information that can identify a specific place (place) like a signboard of a store (for example, a text such as a trade name or an illustration corresponding to a specific trade name) is displayed in an image. An example is shown. For example, the image stream captured by the camera module 310 may be automatically transmitted to the OCR recognition engine 320, and the OCR recognition engine 320 may recognize characters in real time. At this time, the electronic device 1 (110) may determine whether the recognized character is information for identifying a specific place according to the control of the application. When it is determined that the recognized character is information for identifying a specific place, the electronic device 1 (110) transmits the recognized character or the image shown in the first screen example 3010 to the server 150. You can do it. At this time, the server 150 may recognize the more accurate place identifier, extract the place metadata (for example, in the case of a store, trade name, industry, description, etc.) and send it to the electronic device 1 (110). .. At this time, the second screen example 3020 shows an example in which the place metadata provided by the server 150 is displayed on the screen in the form of the upper end notification bar 3021. At this time, a link to a URL related to the corresponding place may be set in the upper end notification bar 3021. On the other hand, the electronic device 1 (110) may analyze the image shown in the first screen example 3010 under the control of the application to determine whether the image is an image of a specific place. In other words, electronic device 1 (110) roughly analyzes the image to determine if it is an image for a particular place, so that a more accurate place identifier is extracted on the server 150. The corresponding image may be transmitted to the server 150.

FIG. 31 is a diagram showing an example in which additional information is provided by recognizing an image code in one embodiment of the present invention. In FIG. 10, a QR code button is displayed as the electronic device 1 (110) recognizes the QR code, and when the user selects the displayed QR code button, the page of the URL including the corresponding QR code is landed. An example was explained. In the embodiment of FIG. 31, the first screen example 3110 shows an example in which an image including a QR code is displayed. At this time, the second screen example 3120 acquires the metadata of the URL included in the recognized QR code from the server 150 along with the recognition of the QR code by the QR code recognition engine 340 and displays it in the form of the upper end notification bar 3121. An example is shown. At this time, a link to the URL included in the QR code may be set in the upper end notification bar 3121. Not only the QR code but also the barcode may be displayed in the form of the upper end notification bar by acquiring the metadata of the related product from the server 150. At this time, the link set in the corresponding upper end notification bar may be a page related to the purchase of the related product.

On the other hand, the metadata displayed in the top notification bar is information included in the page provided by the URL included in the image code, and may include various information such as URL, video, image, description, and the like. ..

Thus, according to embodiments of the present invention, providing meaningful information associated with at least one frame of a video stream continuously captured by a camera and / or a user interface associated with the meaningful information. Is possible.

The system or device described above may be implemented by a hardware component, a software component, or a combination of the hardware component and the software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an ALU (arithmetic logic unit), a digital signal processor, a microcomputer, an FPGA (field program gate array), a PLU (programmable log unit), and a micro. It may be implemented using one or more general purpose computers or special purpose computers, such as a processor or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the OS. The processing device may also respond to the execution of the software, access the data, and record, manipulate, process, and generate the data. For convenience of understanding, it may be described that one processing device is used, but those skilled in the art may include a plurality of processing elements and / or a plurality of types of processing elements. You can understand that. For example, a processor may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

The software may include computer programs, code, instructions, or a combination of one or more of these, configuring the processing equipment to operate at will, or instructing the processing equipment independently or collectively. You may do it. Software and / or data is embodied in any type of machine, component, physical device, virtual device, computer storage medium or device to be interpreted based on the processing device or to provide instructions or data to the processing device. May be converted. The software is distributed on a computer system connected by a network and may be recorded or executed in a distributed state. The software and data may be recorded on one or more computer-readable storage media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The medium may be a continuous recording of a computer-executable program or a temporary recording for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a combination of single or multiple hardware, and is not limited to a medium directly connected to a computer system, but is distributed over a network. It may exist. Examples of media include hard disks, floppy (registered trademark) disks, magnetic media such as magnetic tape, magneto-optical media such as CD-ROMs and DVDs, magneto-optical media such as Floptic disks, and It may include a ROM, a RAM, a flash memory, and the like, and may be configured to record program instructions. In addition, other examples of media include storage media or storage media managed by application stores that distribute applications, sites that supply or distribute various other software, servers, and the like. Examples of program instructions include not only machine language code, such as those generated by a compiler, but also high-level language code, which is executed by a computer using an interpreter or the like.

As described above, the embodiments have been described based on the limited embodiments and drawings, but those skilled in the art will be able to make various modifications and modifications from the above description. For example, the techniques described may be performed in a different order than the methods described, and / or components such as the systems, structures, devices, circuits described may be in a form different from the methods described. Appropriate results can be achieved even if they are combined or combined, or confronted or replaced by other components or equivalents.

Therefore, even different embodiments belong to the attached claims as long as they are equivalent to the claims.

Claims (20)

It is an image processing method
The stage of running the camera module in response to entering image processing mode,
The stage in which the executed camera module continuously receives input of a stream of images and displays them on the screen in order.
The stage of transmitting the input image stream to the recognition engine, and
When there is a recognition result recognized by the recognition engine for the input image stream, the step of displaying the recognition result in order on the screen while the input image stream is being displayed.
Characterized by, including
Image processing method. The recognition engine includes a text recognition engine that recognizes the text included in the input image as the recognition result.
The steps of displaying in order are
Dynamically adjusts the display position of the recognized text based on the position of the text area recognized from the input image.
The image processing method according to claim 1, wherein the image processing method is characterized by the above. The recognition engine includes a text recognition engine that recognizes the text included in the input image as the recognition result.
The steps of displaying in order are
The recognized text is displayed in order on the screen on which the stream of the input image is displayed, and the recognized text is recognized based on the position of the text area in the image in which the text is recognized in the input image. Determine the display position of the text
The image processing method according to claim 1, wherein the image processing method is characterized by the above. The recognition engine includes a text recognition engine that recognizes the text included in the input image as the recognition result.
The image processing method further
The stage of displaying the user interface for user confirmation for the recognized text for the input image stream in order on the screen while the input image stream is being displayed.
including,
The image processing method according to claim 1, wherein the image processing method is characterized by the above. The image processing method further
When user confirmation occurs in the user interface, the stage of displaying the image in which the text is recognized among the input images on the screen, and
A step of further displaying at least one of a user interface for copying the recognized text and a user interface for translating the recognized text on the image displayed on the screen.
including,
The image processing method according to claim 4, wherein the image processing method is characterized in that. The recognition engine includes a text recognition engine that recognizes the text included in the input image as the recognition result.
The steps of displaying in order are
The size of the recognized text is dynamically changed according to the number of characters of the recognized text and displayed on the screen.
The image processing method according to claim 1, wherein the image processing method is characterized by the above. The recognition engine includes an image code recognition engine for recognizing an image code included in the input image.
The steps of displaying in order are
The link to the page corresponding to the recognized image code from the images displayed in order on the screen is used as the recognition result, and the stream of the input images is displayed in order on the displayed screen.
The image processing method according to claim 1, wherein the image processing method is characterized by the above. The image processing method further
If a shooting input occurs while the streams of the input images are being displayed in order, the stage of capturing the image related to the time when the shooting input occurs and displaying it on the screen.
The stage of sending the captured image to the server, and
The stage of receiving the image analysis result of the transmitted image from the server and displaying it in association with the image displayed on the screen.
including,
The image processing method according to claim 1, wherein the image processing method is characterized by the above. The image processing method further
The stage of receiving the detect animation of the object included in the transmitted image from the server, and
The stage of displaying the detect animation on the screen in association with the object,
including,
The image processing method according to claim 8, wherein the image processing method is characterized by the above. The detect animation displays a plurality of feature points extracted from the contour line of the object at positions on the image from which the plurality of feature points are extracted, and at least one of the displayed feature points. Includes an animation that connects the feature points of the part with a line,
The image processing method according to claim 9, wherein the image processing method is characterized in that. It is an image processing method
The stage of receiving the image captured by the camera module of the electronic device via the network,
The stage of generating a detect animation of the object contained in the received image,
The stage of transmitting the generated detect animation to the electronic device,
The stage of generating image search results for the received image, and
The stage of transmitting the generated image search result to the electronic device,
Characterized by, including
Image processing method. The detect animation displays a plurality of feature points extracted from the contour line of the object at positions on the image from which the plurality of feature points are extracted, and at least one of the displayed feature points. Includes an animation that connects the feature points of the part with a line,
The image processing method according to claim 11, characterized in that. The stage of generating the detect animation is
The stage of searching for the objects contained in the received image,
The stage of extracting a plurality of feature points from the contour line of the object,
The step of generating a convex hull of the extracted feature points,
Among the extracted feature points, a step of calculating the center point based on the center coordinates of a predetermined number of outer points constituting the convex hull, and
A step of selecting a feature point closest to an intermediate value between each of the outer outline points and the center point among the feature points of the object.
including,
The image processing method according to claim 11, wherein the image processing method is characterized in that. The step of transmitting the generated detect animation to the electronic device is
Information about the coordinates of the selection points including the outer point, the center point, and the closest feature point, and information about the order in which the selection points are connected by a line are transmitted to the electronic device as the detect animation, or , An animation connecting the selected points in the order is transmitted to the electronic device as the detect animation.
The image processing method according to claim 13, characterized in that. The image search result includes at least one of the object types and names, and further includes at least one of the images, documents, and texts searched in connection with the object.
The image processing method according to claim 11, wherein the image processing method is characterized in that. At least one of the object types and names is displayed on the screen of the electronic device so as to overlap with the image input to the camera module of the electronic device.
At least one of the searched images, documents, and texts contains a link to the corresponding page and is further displayed on the screen of the electronic device in association with the image input to the camera module of the electronic device.
The image processing method according to claim 15, characterized in that. The image processing method further
A step of receiving a signal generated by selecting the type or name of the object displayed on the screen of the electronic device by the user of the electronic device via a network.
In response to the reception of the signal, the stage of generating a text search result using the type or name of the object as a keyword, and
The stage of providing the generated text search result to the electronic device,
including,
The image processing method according to claim 16, wherein the image processing method is characterized in that. The stage of generating the image search result of the received image is
The information searched as a template pre-designed for each type according to the type of information searched in relation to the object included in the received image is realized in the form of a card.
The image processing method according to claim 11, wherein the image processing method is characterized in that. A computer program recorded on a computer-readable storage medium that, when executed, combines with the computer to cause the computer to perform the method according to any one of claims 1-18.
Computer program. A computer-readable storage medium on which a computer program is recorded, which, when executed by the computer, causes the computer to perform the method according to any one of claims 1-18.
Computer-readable storage medium.