WO2019039799A1

WO2019039799A1 - Processing of visual input

Info

Publication number: WO2019039799A1
Application number: PCT/KR2018/009470
Authority: WO
Inventors: 강유훈; 김종택
Original assignee: 네이버 주식회사
Priority date: 2017-08-20
Filing date: 2018-08-17
Publication date: 2019-02-28

Abstract

A technology for processing a visual input is provided. An image processing method according to an embodiment may provide meaningful information associated with at least one frame of an image stream continuously captured through a camera, and/or a user interface associated with the meaningful information.

Description

Processing of visual input

The following description relates to techniques for processing visual input, and more particularly to techniques for processing visual input that can provide a user interface associated with meaningful and / or meaningful information associated with at least one frame of a video stream that is continuously captured via a camera An image processing method and system, and a computer program stored in a computer-readable recording medium for causing a computer to execute an image processing method according to embodiments of the present invention in combination with a computer, and a recording medium therefor.

Various prior art techniques exist for processing visual input such as images. For example, Korean Patent Laid-Open No. 10-2003-0024786 analyzes an entire image taken by a digital camera in relation to text information, recognizes and interprets the information by an OCR (Optical Character Reader) technique, And / or < / RTI > compressed text codes for transmission.

And provides a user interface associated with meaningful and / or meaningful information associated with at least one frame of the video stream that is continuously captured via the camera.

An image processing method and system that can provide a user interface associated with meaningful and / or meaningful information associated with at least one frame of a video stream that is continuously captured via a camera, and in combination with a computer, A computer program stored in a computer readable recording medium for causing a computer to execute an image processing method according to the present invention, and a recording medium therefor.

An image processing method, comprising: driving a camera module in response to entering an image processing mode; Sequentially receiving and sequentially displaying a stream of images through the camera module; Transmitting a stream of the received images to a recognition engine; And displaying the stream of the received images on a screen being displayed sequentially when the recognition result recognized by the recognition engine exists for the stream of the received images. As shown in FIG.

An image processing method comprising: receiving an image captured via a camera module of an electronic device through a network; Generating a detaching animation for an object included in the received image; Transmitting the generated detecming animation to the electronic device; Generating an image search result for the received image; And transmitting the generated image search result to the electronic device.

There is provided a computer program stored in a computer-readable medium for causing a computer to execute the image processing method in combination with a computer.

There is provided a computer-readable recording medium having recorded thereon a program for causing a computer to execute the image processing method.

The computer system comprising: at least one processor configured to execute computer readable instructions, wherein the at least one processor drives a camera module in response to entering an image processing mode, And sequentially transmits the streams of the input images to a recognition engine. When there is a recognition result recognized by the recognition engine for the streams of the received images, And sequentially displaying the streams of the input images on a screen being displayed.

A computer-readable medium having computer-executable instructions for performing the steps of: receiving, via the network, an image captured via a camera module of an electronic device by the at least one processor; Generating a detection result for the object, transmitting the generated detec- tion animation to the electronic device, generating an image search result for the received image, and transmitting the generated image search result to the electronic device And performing operations of the computer system.

And may provide a user interface associated with meaningful and / or meaningful information associated with at least one frame of the video stream being continuously captured through the camera.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.

2 is a block diagram illustrating an internal configuration of an electronic device and a server according to an embodiment of the present invention.

3 is a diagram showing an example of an execution environment of an image processing system according to an embodiment of the present invention.

FIGS. 4 and 5 illustrate examples of providing additional information or additional functions according to OCR recognition in an embodiment of the present invention.

6 to 8 are diagrams illustrating an example of adjusting the font size according to the number of recognized characters in an embodiment of the present invention.

9 is a diagram illustrating an example of providing an additional function according to bar code recognition in an embodiment of the present invention.

10 is a diagram illustrating an example of providing an additional function according to QR code recognition in an embodiment of the present invention.

11 to 15 are views showing examples of providing an image search result in an embodiment of the present invention.

FIG. 16 is a diagram illustrating examples of providing an image search result in an embodiment of the present invention. FIG. 17 is a diagram illustrating an example of limiting the saturation by extracting a main color in an embodiment of the present invention to be.

FIGS. 18 to 20 are views showing examples of providing image search results through a template designed according to a type, in an embodiment of the present invention.

FIGS. 21 to 24 illustrate examples of displaying a detecting animation in an embodiment of the present invention. FIG.

25 is a diagram showing an example of a detection animation in an embodiment of the present invention.

26 is a diagram showing an example of an image processing method according to an embodiment of the present invention.

27 is a view showing another example of the image processing method according to the embodiment of the present invention.

28 is a diagram showing another example of the image processing method according to the embodiment of the present invention.

29 is a flowchart illustrating an example of a method of generating a detaching animation in an embodiment of the present invention.

30 is a diagram showing an example of providing additional information according to place recognition in an embodiment of the present invention.

31 is a diagram showing an example of providing additional information according to recognition of an image code in an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

The image processing method according to embodiments of the present invention can be performed through a computer apparatus such as an electronic apparatus and / or a server to be described later. At this time, a computer program according to an embodiment of the present invention can be installed and driven in the computer apparatus, and the computer apparatus can perform the image processing method according to an embodiment of the present invention under the control of the computer program that is driven . The above-described computer program may be stored in a computer-readable recording medium for execution by a computer in combination with a computer apparatus to perform an image processing method.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. 1 shows an example in which a plurality of

electronic devices

110, 120, 130, 140, a plurality of

servers

150, 160, and a network 170 are included. 1, the number of electronic devices and the number of servers are not limited to those shown in FIG.

The plurality of

electronic devices

110, 120, 130, 140 may be a fixed terminal implemented as a computer device or a mobile terminal. Examples of the plurality of

electronic devices

110, 120, 130 and 140 include a smart phone, a mobile phone, a navigation device, a computer, a notebook, a digital broadcast terminal, a PDA (Personal Digital Assistants) ), And tablet PCs. For example, FIG. 1 illustrates the shape of a smartphone as an example of the first electronic device 110, but in the embodiments of the present invention, the first electronic device 110 transmits the network 170 using a wireless or wired communication method. Or any of a variety of physical computer devices capable of communicating with other

electronic devices

120, 130, 140 and / or servers 150,

The communication method is not limited, and may include a communication method using a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network 170 may include, as well as a short-range wireless communication between the devices. For example, the network 170 may be a personal area network (LAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN) , A network such as the Internet, and the like. The network 170 may also include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, It is not limited.

Each of the

servers

150 and 160 is a computer device or a plurality of computers that communicate with a plurality of

electronic devices

110, 120, 130 and 140 through a network 170 to provide commands, codes, files, Lt; / RTI > devices. For example, the server 150 may be a system that provides a first service to a plurality of

electronic devices

110, 120, 130, 140 connected through a

network

170, 170, and 140 to the first and second

electronic devices

110, 120, 130, and 140, respectively. More specifically, the server 150 may be a computer program that is installed in a plurality of

electronic apparatuses

110, 120, 130, and 140, Mail service, content transfer service, and the like) as a first service to a plurality of

electronic devices

110, 120, 130, 140 in addition to the service for image processing, the information providing service, the messaging service, As another example, the server 160 may provide a service for distributing a file for installing and running the application to the plurality of

electronic devices

110, 120, 130, and 140 as a second service.

2 is a block diagram illustrating an internal configuration of an electronic device and a server according to an embodiment of the present invention. 2 illustrates an internal configuration of the electronic device 1 (110) and the server 150 as an example of the electronic device. Other

electronic devices

120, 130, 140 and server 160 may also have the same or similar internal configuration as electronic device 1 110 or server 150 described above.

The electronic device 1 110 and the server 150 may include memories 211 and 221,

processors

212 and 222,

communication modules

213 and 223 and input /

output interfaces

214 and 224. The memories 211 and 221 may be a computer-readable recording medium and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), and a disk drive. The non-decaying mass storage device such as a ROM and a disk drive may be included in the electronic device 110 or the server 150 as a separate persistent storage device different from the memory 211 or 221. The memory 211 and the memory 221 are provided with an operating system and at least one program code (for example, a program installed in the electronic device 1 (110) and used for a browser or an application installed in the electronic device 1 Code) can be stored. These software components may be loaded from a computer readable recording medium separate from the memories 211 and 221. [ Such a computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, and a memory card. In other embodiments, the software components may be loaded into memory 211, 221 via

communication modules

213, 223 rather than a computer readable recording medium. For example, at least one program may be a computer program installed by files provided by a file distribution system (e.g., the server 160 described above) that distributes installation files of developers or applications, May be loaded into the memory 211, 221 based on the application (e.g., the application described above).

Processors

212 and 222 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the

processors

212 and 222 by the memories 211 and 221 or the

communication modules

213 and 223. For example, the

processor

212, 222 may be configured to execute a command received in accordance with a program code stored in a recording device, such as the memory 211, 221.

The

communication modules

213 and 223 may provide functions for the electronic device 1 110 and the server 150 to communicate with each other through the network 170 and may be provided to the electronic device 1 110 and / May provide a function for communicating with another electronic device (e.g., electronic device 2 120) or another server (e.g., server 160). The request generated by the processor 212 of the electronic device 1 110 according to the program code stored in the recording device such as the memory 211 is transmitted to the server 170 via the network 170 under the control of the communication module 213 150 < / RTI > Conversely, control signals, commands, contents, files, and the like provided under the control of the processor 222 of the server 150 are transmitted to the communication module 223 of the electronic device 110 via the communication module 223 and the network 170 213 to the electronic device 1 (110). For example, control signals, commands, contents, files, and the like of the server 150 received through the communication module 213 can be transmitted to the processor 212 or the memory 211, (The above-mentioned persistent storage device), which may further include a storage medium 110.

The input / output interface 214 may be a means for interfacing with the input / output device 215. For example, the input device may include a device such as a keyboard or a mouse, and the output device may include a device such as a display, a speaker, and the like. As another example, the input / output interface 214 may be a means for interfacing with a device having integrated functions for input and output, such as a touch screen. The input / output device 215 may be composed of the electronic device 1 (110) and one device. The input / output interface 224 of the server 150 may be a means for interfacing with the server 150 or an interface with a device (not shown) for input or output that the server 150 may include. More specifically, when the processor 212 of the electronic device 1 (110) processes the command of the computer program loaded in the memory 211, the configuration is performed using the data provided by the server 150 or the electronic device 2 (120) A service screen or contents can be displayed on the display through the input / output interface 214. [

Also, in other embodiments, electronic device 1 110 and server 150 may include more components than the components of FIG. However, there is no need to clearly illustrate most prior art components. For example, electronic device 1 110 may be implemented to include at least a portion of input / output devices 215 described above, or may be implemented with other components such as a transceiver, Global Positioning System (GPS) module, camera, Elements. More specifically, when the electronic device 1 (110) is a smart phone, the acceleration sensor, the gyro sensor, the camera module, various physical buttons, buttons using a touch panel, input / output ports, A vibrator, and the like may be further included in the electronic device 1 (110).

3 is a diagram showing an example of an execution environment of an image processing system according to an embodiment of the present invention. 3 shows that electronic device 1 110 includes a camera module 310, an OCR recognition engine 320, a barcode recognition engine 330 and a QR code recognition engine 340 and the server 150 includes an image search engine 350 and a detec- tion animation generation engine 360 are shown. The detec- tion animation generation engine 360 may be included in the first electronic device 110 according to the embodiment.

Each of the engines 320 to 360 may be implemented in the form of a software module. For example, the OCR recognition engine 320, the barcode recognition engine 330, and the QR code recognition engine 340 included in the electronic device 1 (110) are provided by an application installed and driven in the electronic device 1 (110) It can be a functional expression. In this case, the processor 212 of the electronic device 110 may perform operations according to the OCR recognition engine 320, the barcode recognition engine 330, and the QR code recognition engine 340 according to an application code. Similarly, the image search engine 350 and the detection animation generation engine 360 included in the server 150 may be implemented in the form of a software module, and may be a functional expression provided by a computer program running on the server 150 . In this case, the processor 222 of the server 150 may perform operations according to the image search engine 350 and the detection animation generation engine 360 according to the code of the computer program.

The OCR recognition engine 320 may recognize one or more characters and / or numbers within the image.

The barcode recognition engine 330 can recognize the barcode in the image.

The QR code recognition engine 340 can recognize the QR code in the image.

The image search engine 350 may receive an image as input and return various search results (image, text, etc.) associated with the image.

Detecting animation generation engine 360 may generate and provide a detaching animation for visually expressing the process of searching for an object in the image. Such detec- ting animation can be utilized as an effect to induce a user's interest while waiting for a search result and give a feeling that the search result is not delayed.

The electronic device 1 110 can drive the camera module 310 and the camera module 310 can be operated without any additional input from the user. To the inputs of the OCR recognition engine 320, the barcode recognition engine 330 and the QR code recognition engine 340. Each of the OCR recognition engine 320, the bar code recognition engine 330 and the QR code recognition engine 340 sequentially analyzes the images (frames) of the input image stream and generates a corresponding object , A number, a bar code, a QR code, etc.).

As described above, the OCR recognition engine 320 may sequentially analyze images included in the image stream to attempt recognition of characters and / or numbers contained within the image, and may return recognized characters and / or numbers . In this case, the electronic device 1 (110) may display the returned characters and / or numbers on the screen, and may provide additional information or additional functions associated with the displayed characters and / or numbers. For example, when a character of a first language is recognized, the electronic device 110 provides a user interface for accessing a translation function capable of translating the recognized first language character into a character of another language . As another example, the electronic device 110 may provide a user interface for accessing a search function that uses the returned characters and / or numbers as a keyword. As another example, the electronic device 1 (110) may automatically perform a search using a character and / or a number, which are returned, and provide the search result.

The barcode recognition engine 330 can sequentially analyze the images included in the image stream to attempt to recognize the barcode included in the image, and to return information on the recognized barcode. In this case, the electronic device 1 (110) may provide additional information or additional functions associated with the returned information. For example, the electronic device 1 (110) searches the information on the returned barcode (for example, information on a book or wine corresponding to the barcode) corresponding to the barcode and returns the search result as additional information . As another example, the electronic device 1 (110) may provide a user interface for accessing information corresponding to the bar code.

The QR code recognition engine 340 can sequentially analyze the images included in the image stream to attempt to recognize the QR code included in the image, and to return information on the recognized QR code. In this case, the electronic device 1 (110) may provide additional information or additional functions associated with the returned information. Similarly to the case of the barcode, the electronic device 110 provides information (e.g., information corresponding to the URL included in the QR code) corresponding to the recognized QR code as additional information, or provides information corresponding to the recognized QR code And may provide a user interface for accessing information.

When the electronic device 1 110 enters the image processing mode, the electronic device 1 110 drives the camera module 310 without any additional input from the user, and the image provided through the camera module 310 For each of the images in the stream, the search for objects such as letters, numbers, barcodes, QR codes, etc. can be performed automatically, and additional information or additional functions related to the searched objects can be automatically provided.

On the other hand, the electronic device 1 (110) generates a user input through a predetermined user interface such as a selection of a user's photographing button (for example, a user touches an area of a photographing button displayed on the touch screen in a touch screen environment) Can be monitored. Referring to FIG. 3, the process 370 monitors whether a user input is generated and, when a user input occurs, the captured image is transmitted to the server 150 according to user input. 2, the electronic device 1 110 may transmit the captured image via the network 170 to the server 150 using the communication module 213, and the server 150 may transmit the captured image via the network 170 to the server 150, May receive the captured image transmitted via the network 170 via the communication module 223.

The server 150 may provide the delivered image to the image search engine 350 and the detection animation generation engine 360, respectively.

As described above, the image search engine 350 may receive the captured image captured by the first electronic device 110, and may return various search results associated with the image. For example, the image search engine 350 may recognize an object included in the image and search for and return an image, document, text, or the like related to the recognized object. As a more specific example, when a puppy included in the image is recognized and the puppy of the recognized dog is analyzed as a 'retriever', a search result such as an image or a document related to the 'retriever' can be generated and returned. The server 150 may transmit the retrieved search result to the electronic device 110 through the network 170 and the electronic device 110 may provide the search result to the user. According to the embodiment, the electronic device 1 110 not only receives the image but also the time when capturing the image, the current position of the electronic device 110, information of the user of the electronic device 110, and the like to the server 150 It may be further transmitted. In this case, the server 150 may provide search results based on at least one of location, time, and user information. For example, a search result related to a user's current position or a search result related to time among various search results associated with an image can obtain a priority for exposure of the search result.

The detec- tion animation generation engine 360 may generate a detec- tion animation for visually expressing a process of capturing an image captured by the electronic device 110 and receiving the received image as input and searching for an object in the image. In this case, the server 150 may transmit the generated detecting animation to the first electronic device 110 via the network 170 and may associate the detected animation with the corresponding image for a predetermined time The user of the electronic device 1 110 waits for the search result (the search result returned through the image search engine 350 described above and provided to the electronic device 1 110 at the server 150) And can be utilized as an effect to give an impression that the search result is not delayed. Such a detecting animation may basically consist of a plurality of points of a position related to an object to be searched in the image and a line connecting these points and a representation of the lines connecting the points is displayed as an animation effect . Further, according to the embodiment, the thickness, size, brightness, color, etc. of dots and lines may be changed to provide an additional animation effect. In addition, embodiments may be considered in which the lines formed by the lines connecting the points and the points are displayed in different colors to give a three-dimensional effect or curved lines connecting the points. This detecting animation will be described in more detail later.

4 shows screen examples 410 to 440 of the electronic device 1 (110). 3, an image stream captured through the camera module 310 is automatically transmitted to the OCR recognition engine 320, and the OCR recognition engine 320 real-time recognizes characters As shown in FIG.

At this time, in the second screen example 420, as the recognition is completed, the color of the recognized character is changed and displayed on the image, and the T (TEXT) character recognition button is exposed.

In addition, the third screen example 430 shows an example in which functions related to recognized characters such as copying, translation, and reading of recognized characters are provided by selecting the T character recognition button. For example, the copy function may be a function for copying recognized characters to the clipboard. In addition, the translation function may be a function for translating a recognized first language character into a second language character. Further, the reading function may be a function of reading the recognized first character, or may be a function of generating and outputting audio corresponding to the first character.

The fourth screen example 440 shows an example in which characters of the first language recognized as the user selects the translation function are translated and displayed as characters of the second language. At this time, the detail view function displayed in the fourth screen example 440 may provide additional contents such as a language dictionary search result for the recognized first language character, a search result using the recognized first language character as a keyword, or the like It may be a function for landing on a separate translation result page. Further, if the recognition is not performed correctly or the translation result is not the desired result, a handwriting search function for proceeding with the search using the handwriting recognition function button may be further provided.

In the embodiment of FIG. 4, an example has been described in which an additional function is provided for all recognized characters. However, according to an embodiment, some of the recognized characters may be selected, and a corresponding additional function may be provided have.

5 shows screen examples 510 to 530 of the electronic device 1 (110). At this time, the first screen example 510 shows an example in which an image is displayed on the screen before recognition of text (characters and / or numbers).

In addition, the second screen example 520 shows an example in which recognized text is displayed at a position similar to the position of the text in the image as the text is recognized.

The third screen example 530 shows an example in which the recognized text is reconstructed and displayed so that the user can easily understand it. At this time, the third screen example 530 shows an example in which the image is darkened so that the recognized text can be expressed better. In the third screen example 530, an example of providing additional functions such as a copying function, a translation function, and a reading function in association with the recognized text is described.

At this time, the texts displayed in the second screen example 520 and the third screen example 530 may have an animation effect such as an animation effect such as a color change, a three-dimensional change in a two-dimensional frame in which texts are displayed, It may be displayed together.

FIG. 6 shows an example of displaying text recognized as 80 px size when the recognized number of characters is 1 to 6 characters.

In addition, FIG. 7 shows an example of displaying a text recognized as a 60px size when the recognized number of characters is 7 to 40 characters.

FIG. 8 shows an example of displaying a text recognized as a 40px size when the recognized number of characters is 41 or more.

As described above, in displaying the recognized text according to the number of characters of the text recognized in the image, the electronic device 1 (110) automatically adjusts the size of the displayed character.

9 shows screen examples 910 and 920 of the electronic device 1 (110). The first screen example 910 shows an example in which an image including a bar code is displayed and the second screen example 920 shows an example in which a bar code button is exposed as a bar code is recognized by the bar code recognition engine 330 . When a user selects an exposed barcode button, product information (for example, book information, wine information, and the like) corresponding to the barcode may be provided.

10 shows screen examples 1010 and 1020 of the electronic device 1 (110). In the first screen example 1010, an image including a QR code is displayed. In the second screen example 1020, a QR code button is exposed as the QR code is recognized by the QR code recognition engine 340 For example. When a user selects an exposed QR code button, the page of the URL included in the QR code can be landed.

11 shows screen examples 1110 to 1130 of the electronic device 1 (110). The first screen example 1110 shows an example in which an image is displayed, and the second screen example 1120 shows an example in which an object is searched in an image. At this time, the image can be transmitted to the server 150 as the user presses the photographing button. In the server 150, the image retrieval engine 350 and the detec- tion animation generation engine 360 can perform image retrieval and detection animation Can be generated. The second screen example 1120 receives a detecting animation composed of lines connecting the plurality of dots and points from the server 150 and displays it on the screen to visually detect that the dog is searching for a face As shown in FIG. Also, the third screen example 1130 shows an example in which "dog" and "labrador retriever" are displayed as text information (associated keywords) related to the searched object, and image search results for the image are further displayed. The text information and the image search result may be provided through the server 150. At this time, when an area displaying "dog" or "labrador retriever" displayed as text information related to the searched object is selected for the user, a text search result using the corresponding text information as a keyword may be provided to the user. In addition, when each of the image search results is selected by the user, a page corresponding to the search result may be landed.

12 shows screen examples 1210 to 1240 of the electronic device 1 (110). The first screen example 1210 shows an example in which an image is displayed and the second screen example 1220 and the third screen example 1230 show a detaching animation composed of lines connecting a plurality of dots and points, And an example of a process of visually informing the user that the fish is being searched. Also, the third screen example 1240 shows an example in which "aquarium fish" and "Asian arowana" are displayed as text information (associated keywords) related to the searched object, and image search results for the image are further displayed. In this case, when an area displaying "aquarium fish" or "Asian aurora" displayed as text information related to the searched object is selected by the user, a text search result using the text information as a keyword can be provided to the user. In addition, when each of the image search results is selected by the user, a page corresponding to the search result may be landed.

FIGS. 13 to 15 also show examples of processes for providing image search results as shown in FIGS. 11 and 12. FIG. FIG. 13 illustrates a process of recognizing an object through detec- tion along a contour line of a flower, providing a detec- tion animation and an image search result, FIG. 14 illustrates a process of recognizing an object by detecting a cat face along an outline, FIG. 15 illustrates the process of providing an animation and an image search result, FIG. 15 illustrates a process of recognizing an object by detecting an Eiffel tower along an outline line, and providing a detecting animation and an image search result.

FIG. 16 is a diagram illustrating examples of providing an image search result in an embodiment of the present invention. FIG. 17 is a diagram illustrating an example of limiting the saturation by extracting a main color in an embodiment of the present invention to be. FIG. 16 shows an example of extracting a main color of an image using an auto technique and utilizing the extracted main color for exposure of a related keyword or an image search result. In this case, the saturation (S) and the brightness (B) values may be limited to a range of 50 to 70% in the hue-saturation-brightness (HSB) value in consideration of the visibility of the text. FIG. 17 shows an example in which the main color is extracted using a color picker to limit the saturation value to 50%.

18 to 20 are views showing examples of providing an image search result in an embodiment of the present invention. Figures 18 to 20 illustrate the results of a method of determining whether a person is a member of a group of persons, such as "person_domain", "person_group", "person_outside", " And " Domestic Place ", and the like, the specific search result of the image search result is implemented in the form of the correct answer card through the template designed in advance according to various types. In this case, the saturation or brightness value of the HSV (Hue-Saturation-Brightness) value may be limited to 50 to 70% in view of the visibility of the text.

The method of generating the detection animation is as follows. For example, a detection animation generation method may be performed by the detection animation generation engine 360 described above.

(1) Preprocessing process: Performing the pre-processing required for image search such as grayscale, blurring, and edge detection.

(2) Object Detection: The process of searching for an object in the input image and creating a bounding box containing the object. In general, a well-known object search technique can be used to search for an object, and the following process can be performed on the generated bounding box area.

(3) The process of extracting meaningful feature points from the outline of an object. For example, a predetermined number of feature points (e.g., several hundreds) can be extracted using the FAST algorithm.

(4) The process of generating a convex hull for extracted minutiae. For example, a convex polygon including all of the feature points may be generated as a convex hole, and the generation of the convex hole may be generated through a known algorithm.

(5) If the point constituting the convex hole does not reach a predetermined number (for example, 6), proceeding to an additional step to add points until the necessary number is reached. For example, it can be used as a point for constructing a convex hole by selecting a feature point closest to the middle between two points constituting a convex hole.

(6) The process of constructing outline points (outline points) forming outlines with the points selected in (4) and (5) and calculating the center coordinates (center point) of outline points. For example, a point of each coordinate (average of X values and average of Y values) can be calculated as a center point.

(7) For each of the outer points, a process of selecting a feature point closest to the middle value between the outer point and the center point

And repeating steps (6) and (7) when it is desired to add a line to the inside of the object (8). It can be omitted if only one level of lines are connected.

(9) The process of returning the coordinates and animation sequence of points, or returning the generated animation.

The process of generating the detecting animation according to the processes (1) to (9) may be modified into various forms according to the design as one embodiment.

For example, FIG. 21 shows a detecting animation having a triangle structure by connecting points of a convex hole and a depth 3, which are composed of five points. More specifically, FIG. 21 shows a case in which pentagonal dots are formed and connected to an outline of an object which is a convex hole, two dots of depth are formed and connected to pentagonal dots, a left and a right of two dots of depth are connected, An example of generating a detecting animation form by vertically connecting two dots of depth and three dots of depth is shown. At this time, an example in which the depth 2 dots are formed at one-third of the line connecting the center of the connecting line between the pentagonal dots and the center point (depth 3 dots) is shown.

In addition, FIG. 22 shows a detecting animation having a triangle and a quadrilateral structure by connecting points of a convex hole and a depth 3, which are composed of nine dots. In Fig. 22, the color difference between the contour dot (nine points constituting the convex hole) and the center dot (center point) is not less than a predetermined value (for example, 20 or more from the B value of the starting point RGB value of the contour dot And a dot having the closest distance between the dots of the upper and lower depths is connected to generate a detecting animation form.

In FIGS. 23 and 24, examples of varying the color of the triangular structure to give a three-dimensional effect and simultaneously controlling the transparency are shown. In other words, FIG. 23 shows an example in which the transparency in the triangular structure is higher in FIG.

25 is a diagram showing an example of a detection animation in an embodiment of the present invention. FIG. 25 is a view showing a result of the detection of the animation of the Epithelial Top described with reference to FIG. 15, as a result of repeating display of lines connecting points and dots as shown in FIG. 25, It can be utilized as an effect for inducing the user's interest while waiting for the search result and giving the user the feeling that the search result is not delayed. At this time, as described above, the detecting animation may change the thickness, size, brightness, color, etc. of dots and lines to give additional animation effects. As described in FIGS. 23 and 24, By applying differently, it is possible to determine the extent to which the image is displayed by giving a three-dimensional effect or adjusting the transparency of the color.

26 is a flowchart showing an example of an image processing method according to an embodiment of the present invention. The image processing method according to the present embodiment can be performed by a computer apparatus such as the above-described electronic apparatus 1 (110). For example, the processor 212 of the electronic device 1 110 may be implemented to execute a control instruction according to code of an operating system included in the memory 211 or code of at least one computer program. Here, the processor 212 controls the electronic device 1 (110) to perform the steps 2610 to 2640 included in the image processing method of FIG. 26 according to the control command provided by the code stored in the electronic device 1 (110) It is possible to control the device 1 (110).

In step 2610, the computer device may drive the camera module in response to entering the image processing mode. In one example, an application installed on a computer device may provide a user interface for entering the image processing mode to the user. When an input to the user's user interface occurs, the computer device can drive the camera module according to the entering image processing mode according to the generated input.

In step 2620, the computer device may sequentially receive and sequentially display a stream of images through the driven camera module. For example, when a camera is operated in a smartphone, an image input through a camera and displayed in real time on a screen of a smartphone may correspond to a stream of such images.

In step 2630, the computer device may deliver a stream of received images to the recognition engine. The camera module can deliver a continuous stream of images to the recognition engine on a continuous and real-time basis, and the recognition engine can analyze the stream of images to generate the recognition results desired by the recognition engine. For example, in the embodiment of FIG. 3, electronic device 1 110 includes a text recognition engine, such as OCR recognition engine 320, and image code recognition engines, such as bar code recognition engine 330 and QR code recognition engine 340, An example of including this has been described.

In step 2640, when there is a recognition result recognized by the recognition engine for the stream of the received images, the computer device may further display the stream of the received images on the screen being displayed sequentially. For example, the recognition engine may include a text recognition engine that recognizes text included in the images received as recognition results.

At this time, if the recognition result recognized by the text recognition engine exists, the computer device can dynamically adjust the display position of the recognized text based on the position of the text area recognized in the input images in step 2640 have. For example, the first screen example 410 of FIG. 4 shows an example in which recognized text is displayed at a location of an area containing text in an image. As another example, in the second screen example 520 of FIG. 5, an example in which recognized text is displayed at a position similar to a position in the image of the text as the text is recognized is described. For example, as the computer device such as a smart phone shakes in the user's hand, the position of the same text area in the received images may be continuously changed. In this case, the computer device can dynamically adjust the display position of the text so that the recognized text can be displayed at the tracked position by tracking the position of the text area as the position of the text area is changed.

In addition, in step 2640, the computer device displays the recognized text on the screen sequentially displaying the streams of the input images, wherein the text of the input images is based on the position of the text area in the recognized image The display position of the recognized text can be determined. For example, as described above, the computer device dynamically adjusts the display position of the recognized text as the position of the text area is changed, and when the recognition of the text is finally completed, The text can be displayed. In this case, the display position of the recognized text may be fixed even if the position of the text area continuously changes in the received images or other images without text are continuously input.

In addition, the computer device may further display a stream of input images on a screen being displayed sequentially, through a user interface for user confirmation of the recognized text with respect to a stream of inputted images. For example, after the recognition of the text is finally completed by the text recognition engine, the computer device can finally process the process of receiving the user's confirmation of the recognized text. For example, the second screen 420 shown in FIG. 4 shows an example of displaying a user interface for confirming a recognized text such as a character recognition button. At this time, a stream of images may still be displayed in real time on the screen of the computer device.

At this time, when a user confirmation is made through a user interface for confirming the recognized text to the user, the computer device can display an image displaying the text of the received images on the screen. For example, if the text recognized by the user is recognized through the first image of the input images, the computer device may display the first image that has already been displayed on the screen instead of displaying the stream of images on the screen. The computer device may further display at least one of a user interface for copying the recognized text on the image displayed on the screen and a user interface for the translation of the recognized text. For example, FIGS. 4 and 5 show examples of user interfaces for copying and translating recognized text, such as a 'Copy' button and a 'Translation' button.

Also, the computer device may dynamically change the size of the recognized text according to the number of characters of the recognized text, and display the changed size on the screen. For example, in FIGS. 6 to 8, an example has been described in which the recognized text is dynamically reduced and displayed on the screen as the number of characters in the recognized text increases.

Further, as another example, the recognition engine may include an image code recognition engine for recognizing the image code included in the received images. In this case, the computer device displays a stream of images sequentially received as a recognition result of the link to the page corresponding to the recognized image code in the images sequentially displayed on the screen in step 2640, can do. For example, FIG. 9 shows an example of displaying a link to a page corresponding to a recognized bar code through a 'barcode' button. In FIG. 10, a link to a page corresponding to the recognized QR code is referred to as a 'QR code' Button is displayed.

According to an embodiment, the recognition engine may comprise a plurality of recognition engines, such as a text recognition engine and an image code recognition engine, in which case a stream of images input via the camera module may be input to each of a plurality of recognition engines .

The steps 2610 to 2640 described above may be performed in such a manner that the computer device automatically recognizes text, image codes, and the like through a stream of images input through the camera before the user's photographing through the camera is performed An embodiment is described. In other words, in response to entering the image processing mode, even if the user does not select an image at a specific point in time by pressing the photographing button, the recognition result for the text or image code can be automatically provided have.

On the other hand, when the user presses the photographing button to select a specific image, the computer device can provide the user with a function different from the previous embodiment.

27 is a view showing another example of the image processing method according to the embodiment of the present invention. The image processing method according to the present embodiment can also be performed by a computer apparatus such as the electronic apparatus 1 (110) described above. Steps 2710 through 2750 of FIG. 27 may be performed after the step 2620 of FIG. 26, when a shooting input occurs, and when steps of FIG. 27 are performed, steps 2630 and 2640 ) May be omitted.

In step 2710, when a shooting input occurs during the sequential display of a stream of input images, the computer device may capture an image associated with the shooting input occurrence time and display the captured image on the screen. This step 2710 may refer to a process of taking a picture according to the occurrence of a photographing input by a user.

In step 2720, the computer device may send the captured image to the server. In one example, the server may correspond to a computer device such as the server 150 described above, and the captured image may be transmitted to the server via the network 170. [

In step 2730, the computer device may receive a detaching animation for an object included in the image transmitted from the server. For example, the detection animation may include an animation that displays a plurality of feature points extracted from an outline of an object at a position on an extracted image of a plurality of feature points, and connects at least some feature points of the displayed feature points with lines .

In step 2740, the computing device may display the detection animation in association with the object. Examples of displaying the detection animation in association with the object on the screen have been described with reference to Figs. 11 to 15, and Figs. 21 to 25. Fig.

In step 2750, the computer device receives the image analysis result of the image transmitted from the server and displays the image analysis result in association with the image displayed on the screen. The image analysis result may include a kind and / or name of an object included in the image, and may further include at least one of images, documents, and text retrieved from the server in association with the object.

28 is a diagram showing another example of the image processing method according to the embodiment of the present invention. The image processing method according to the present embodiment can be performed by a computer apparatus such as the server 150 described above. For example, the processor 222 of the server 150 may be implemented to execute control instructions in accordance with the code of the operating system or the code of at least one computer program that the memory 221 contains. Here, the processor 222 determines whether the server 150 performs the steps 2810 through 2850 included in the image processing method of FIG. 28 in accordance with the control command provided by the code stored in the server 150 Can be controlled.

In step 2810, the computer device may receive the captured image via the camera module of the electronic device over the network. Here, the electronic device may correspond to the electronic device 1 (110) described above, and the received image may correspond to the image transmitted in step 2720 of FIG. In other words, when the electronic device 1 (110) transmits the captured image through the network 170 according to the occurrence of the user's shooting input, the server 150 can receive the image in step 2810.

In step 2820, the computing device may generate a detaching animation for the object that the received image contains. As described above, the detection animation may include an animation that displays a plurality of feature points extracted from an outline of an object at a position on the extracted image of the plurality of feature points, and connects at least some of the feature points of the displayed feature by lines. have.

In step 2830, the computer device may transmit the generated detecting animation to the electronic device. In this case, as described through step 2730 of FIG. 27, the electronic device may receive the detection animation and display the received detection animation in association with the object on the screen, as in step 2740.

In step 2840, the computer device may generate an image search result for the received image. The image search result may include the type and / or name of an object included in the image, and may further include an image, document and / or text retrieved in association with the object.

In step 2850, the computer device may send the generated image search results to the electronic device. At this time, the type and / or the name of the object may overlap with the image input through the camera of the electronic device and displayed on the screen of the electronic device. Further, the retrieved image, document and / or text may be further displayed on the screen of the electronic device in association with the image input through the camera module of the electronic device, including a link to the corresponding page. For example, Figure 12 shows the types and names of recognized objects, such as 'ornamental language' and 'Asian arowana', and further associates images, documents and / or text with images as a result of an Internet search through images Is displayed.

In addition, the computer device may implement the information retrieved through the template designed in advance for each type according to the type of the information retrieved in association with the object included in the received image, in the form of a card, and provide the retrieved information to the electronic device. For example, Figs. 18 to 20 illustrate examples of the types of information (person_domain, person_group, person_outside, bag_ animal, bag_furniture, bag_wine, spot, area, domestic_place, etc.) There is a pre-designed template, and information retrieved through a template of the type according to the type of the retrieved information is implemented and provided in the form of a card.

At this time, when the type and / or the name of the object displayed on the electronic device is selected (for example, the user touches the area where the type and / or the name of the object is displayed in the touch screen environment with the finger) The generated signal can be transmitted to the server via the network. In this case, the computer device can receive the corresponding signal, and in response to receiving the signal, the text search result can be generated by using the type or name of the object as a keyword. In addition, the computer device can provide the generated text search result to the electronic device. In other words, the user of the electronic device can receive sequentially the text search result on the text obtained through the image, in addition to the image search result on the image.

29 is a flowchart illustrating an example of a method of generating a detaching animation in an embodiment of the present invention. The steps 2910 to 2950 included in the method of the present embodiment may be performed in step 2820 of Fig.

In step 2910, the computer device may search for an object that the received image contains. For example, the computer device performs a preprocessing operation required for image search such as grayscale, blurring, and edge detection on a received image, and then searches for an object in the image and creates a bounding box including the object Can be generated. Searching for these objects can generally be done using well-known object search techniques.

In step 2920, the computing device may extract a plurality of feature points from the contour of the object. For example, a predetermined number of feature points (e.g., several hundreds) can be extracted using a Feature from Accelerated Segment Test (FAST) algorithm.

In step 2930, the computer device may generate a convex hull for the extracted feature points. For example, a convex hole can be generated from minutiae extracted through the convex hole algorithm. If the point for constructing the convex hole does not reach the predefined number, the feature points may be further extracted.

In step 2940, the computer device may calculate a center point based on the center coordinates of a predetermined number of outline points constituting the convex hole among the extracted minutiae points. For example, the points of the outer points constituting the convex hole (the average of the X coordinate value and the average of the Y coordinate value) can be calculated centrally.

In step 2950, the computer device may select feature points that are closest to the midpoint between each of the outline points and the midpoint among the feature points of the object. The selected minutiae can be used again as outer minus points to obtain the center point, and then the inner minus points can be added by selecting the minutiae points between the outer points and the center point. For example, FIG. 21 shows an example in which a detecting animation having a triangular structure is formed by connecting points of a convex hole and a depth 3 which are composed of five points.

The generated detection animation may be transmitted to the electronic device as in step 2830 of FIG. 28 and may be displayed on the screen of the electronic device in association with the object in the electronic device, such as in step 2740 of FIG. 27 . At this time, the computer device may transmit information on the coordinates of the selection points including the outer points, the center point and the closest minutiae points, and the order of connecting the selection points by lines to the electronic device as a detecting animation. In this case, the electronic device may display a line through the information of the coordinates of the selected points, and may display an animation connecting the selected points with lines according to the information on the sequence. The computer device may also transmit the animation itself connecting the selection points in this order to the electronic device as a detection animation. In this case, the electronic device can display the detection animation by reproducing the animation in association with the object.

In the process of providing the image processing result to the user, such a detection animation shows that an analysis is performed on an object of an image that the user requests to search. Thus, while the user waits for an image search result, You can give the impression that the result is not delayed.

30 is a diagram showing an example of providing additional information according to place recognition in an embodiment of the present invention. 30 shows screen examples 3010 and 3020 of the electronic device 1 (110). The first screen example 3010 shows an example in which information (for example, a text such as mutual or a picture corresponding to a specific name, etc.) capable of identifying a specific place such as a signboard of a shop is displayed on the image. For example, an image stream captured through the camera module 310 may be automatically delivered to the OCR recognition engine 320 and recognized by the OCR recognition engine 320 in real time. At this time, the electronic device 1 (110) can determine whether the recognized character is information for identifying a specific place according to the control of the application. If it is determined that the recognized character is information for identifying a specific place, the electronic device 110 may transmit the recognized character or the image shown in the first screen example 3010 to the server 150. [ At this time, the server 150 can obtain a more accurate place identifier, and can extract metadata of the place (for example, mutual, business type, description, etc., in the case of a store) and transmit it to the electronic device 1 (110). In this case, the second screen example 3020 shows an example where the metadata of the place provided from the server 150 is displayed on the screen in the form of an upper notification bar 3021. At this time, a link to a URL related to the place may be set in the upper notification bar 3021. [ On the other hand, the electronic device 1 (110) may analyze the image displayed in the first screen example (3010) according to the control of the application to determine whether the image is an image for a specific place. In other words, the electronic device 1 110 roughly analyzes the image to determine whether or not the image is a specific place, and then transmits the image to the server 150 so that an identifier of a more accurate place can be extracted through the server 150 Lt; / RTI >

31 is a diagram showing an example of providing additional information according to recognition of an image code in an embodiment of the present invention. 10, when the QR code button is exposed as the electronic device 1 (110) recognizes the QR code and the user selects the exposed QR code button, an example in which the page of the URL included in the QR code is landed . In the embodiment of FIG. 31, the first screen example 3110 shows an example in which an image including a QR code is displayed. At this time, the second screen example 3120 obtains the metadata of the URL included in the recognized QR code as the QR code is recognized by the QR code recognition engine 340 from the server 150, As shown in Fig. At this time, a link to a URL included in the QR code may be set in the upper notification bar 3121. [ The metadata of the related product can be obtained from the server 150 in the form of an upper notification bar in addition to the QR code as well as the bar code. At this time, the link set in the upper notification bar may be a page associated with the purchase of the related product.

Meanwhile, the metadata displayed in the upper notification bar is information included in a page provided through a URL included in the image code, and may include various information such as a URL, a moving image, an image, a description, and the like.

As such, embodiments of the present invention may provide a user interface associated with meaningful and / or meaningful information associated with at least one frame of a video stream that is continuously captured through a camera.

The system or apparatus described above may be implemented as a hardware component, a software component or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device As shown in FIG. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The medium may be one that continues to store computer executable programs, or temporarily store them for execution or download. In addition, the medium may be a variety of recording means or storage means in the form of a combination of a single hardware or a plurality of hardware, but is not limited to a medium directly connected to a computer system, but may be dispersed on a network. Examples of the medium include a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floptical disk, And program instructions including ROM, RAM, flash memory, and the like. As another example of the medium, a recording medium or a storage medium managed by a site or a server that supplies or distributes an application store or various other software to distribute the application may be mentioned. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims

An image processing method comprising:

Driving the camera module in response to entering the image processing mode;

Sequentially receiving and sequentially displaying a stream of images through the camera module;

Transmitting a stream of the received images to a recognition engine; And

Further comprising the step of, when the recognition result recognized by the recognition engine exists for the stream of the input images, further displaying the recognition result on the screen being displayed sequentially

The image processing method comprising the steps of:
The method according to claim 1,

Wherein the recognition engine includes a text recognition engine that recognizes a text included in the input images as the recognition result,

Wherein the step of displaying further comprises:

And dynamically adjusting a display position of the recognized text based on a position of a text area recognized in the input images.
The method according to claim 1,

Wherein the recognition engine includes a text recognition engine that recognizes a text included in the input images as the recognition result,

Wherein the step of displaying further comprises:

Wherein the display unit displays the recognized text on a screen in which a stream of the input images is sequentially displayed, and displays a display position of the recognized text based on a position of a text area in an image in which the text is recognized, The image processing method comprising the steps of:
The method according to claim 1,

Wherein the recognition engine includes a text recognition engine that recognizes a text included in the input images as the recognition result,

Further comprising displaying a user interface for user identification of the recognized text on the stream of input images on a screen being displayed sequentially,

The image processing method further comprising:
5. The method of claim 4,

Displaying a recognized image of the input images on a screen when a user confirmation is made through the user interface; And

Displaying at least one of a user interface for copying the recognized text and a user interface for translating the recognized text on an image displayed on the screen

The image processing method further comprising:
The method according to claim 1,

Wherein the recognition engine includes a text recognition engine that recognizes a text included in the input images as the recognition result,

Wherein the step of displaying further comprises:

And dynamically changing the size of the recognized text according to the number of characters of the recognized text, and displaying the changed text on the screen.
The method according to claim 1,

Wherein the recognition engine includes an image code recognition engine for recognizing an image code included in the input images,

Wherein the step of displaying further comprises:

Further comprising a link to a page corresponding to the recognized image code in the images sequentially displayed on the screen, on the screen being displayed the streams of the input images sequentially as the recognition result .
The method according to claim 1,

Capturing an image associated with a time point at which the photographing input is generated and displaying the captured image when the photographing input occurs while sequentially displaying the stream of the input images;

Transmitting the captured image to a server; And

Receiving image analysis results of the transmitted image from the server, and displaying the image analysis result in connection with the image displayed on the screen

The image processing method further comprising:
9. The method of claim 8,

Receiving a detaching animation for an object included in the transmitted image from the server; And

Displaying the detecming animation on the screen in association with the object

The image processing method further comprising:
10. The method of claim 9,

Wherein the detecting animation includes an animation for displaying a plurality of feature points extracted from an outline of the object at a position on the image from which the plurality of feature points are extracted and connecting at least some feature points of the displayed feature points by a line The image processing method comprising the steps of:
An image processing method comprising:

Receiving an image captured via a camera module of an electronic device via a network;

Generating a detaching animation for an object included in the received image;

Transmitting the generated detecming animation to the electronic device;

Generating an image search result for the received image; And

Transmitting the generated image search result to the electronic device

The image processing method comprising the steps of:
12. The method of claim 11,

Wherein the detecting animation includes an animation for displaying a plurality of feature points extracted from an outline of the object at a position on the image from which the plurality of feature points are extracted and connecting at least some feature points of the displayed feature points by a line The image processing method comprising the steps of:
12. The method of claim 11,

Wherein the generating the detaching animation comprises:

Searching for an object included in the received image;

Extracting a plurality of feature points from an outline of the object;

Generating a convex hull for the extracted minutiae;

Calculating a center point based on a center coordinate of a predetermined number of outline points constituting the convex hole among the extracted minutiae; And

Selecting a feature point closest to an intermediate value between each of the outline points and the center point among the feature points of the object

The image processing method comprising the steps of:
14. The method of claim 13,

Wherein the transmitting the generated detecming animation to the electronic device comprises:

Information on coordinates of the selection points including the outer points, the center point and the closest feature points, and the order of connecting the selection points by a line to the electronic device as the detecting animation, And transmitting an animation linking the points in the above order to the electronic device as the detecting animation.
12. The method of claim 11,

Wherein the image search result includes at least one of a type and a name of the object, and further includes at least one of an image, a document, and a text retrieved in association with the object.
16. The method of claim 15,

At least one of a type and a name of the object overlaps with an image input through the camera module of the electronic device and is displayed on a screen of the electronic device,

Wherein at least one of the searched image, document, and text includes a link to a corresponding page, and is further displayed on a screen of the electronic device in association with an image input through a camera module of the electronic device .
17. The method of claim 16,

Receiving, via a network, a signal generated as a type or name of the object displayed on the screen of the electronic device is selected by a user of the electronic device;

Generating a text search result using a type or name of the object as a keyword in response to receiving the signal; And

Providing the generated text search result to the electronic device

The image processing method further comprising:
12. The method of claim 11,

Wherein the generating the image search result for the received image comprises:

And the information retrieved through the template designed in advance according to the type of information retrieved in relation to the object included in the received image is implemented in the form of a card.
A computer program stored on a computer readable medium for causing a computer to execute the method of any one of claims 1 to 18 in combination with the computer.
A computer-readable recording medium storing a program for causing a computer to execute the method according to any one of claims 1 to 18.