KR20200127928A - Method and apparatus for recognizing object of image in electronic device - Google Patents

Abstract

The present invention relates to a method and a device for recognizing a specific object in an image in an electronic device. The method for recognizing a specific object in an image in an electronic device can comprise the following steps of: displaying at least one image; detecting at least one gesture; selecting a recognition function associated with at least one object in the at least one image according to the gesture; recognizing the at least one object by using the selected recognition function. Various other embodiments are possible.

Description

A method and apparatus for recognizing a specific object in an image in an electronic device {METHOD AND APPARATUS FOR RECOGNIZING OBJECT OF IMAGE IN ELECTRONIC DEVICE}

Various embodiments of the present invention relate to a method and apparatus for recognizing a specific object in an image in an electronic device.

Conventionally, a touch screen interface is used for providing basic functions related to image capture by a camera through a graphical user interface (GUI). For example, in order to use a photo shooting mode, a video shooting mode, a panoramic photo shooting mode, etc., a shooting mode by tapping or touching a specific icon or specific item from one or more icons or lists representing the shooting mode. You can select and run it. This method is for selecting a shooting mode of the camera or a basic function for shooting, and other functions may be driven through a touch screen interface in addition to the above example. For example, in addition to the purpose of taking a picture through an image sensor mounted on an electronic device such as a portable terminal and a smart phone, the user is used for reading specific information on a subject in a preview state.

In particular, applications such as barcode recognition, two-dimensional code, color-based code, business card recognition, face recognition, etc., have been widely used in recent years. In general, a touch screen interface may be used to drive a separate application according to the type of object to be recognized, or to select a type of a specific recognition module in the same application.

An application for recognizing a business card, barcode or QR (Quick Response) code in an electronic device acquires an image corresponding to a specific business card, barcode and QR code through an image sensor, and then extracts the barcode and QR code from the image. Information included in the extracted code can be displayed.

The method of extracting the area of the object to be recognized from the entire image acquired through the image sensor takes a lot of computation and time, and may increase the possibility of extracting the wrong area. Accordingly, in order to increase the recognition rate and image processing efficiency, a recognition area can be created so that recognition is performed only within a predetermined limited area of an image, and the range of the corresponding recognition area can be displayed on the display. As an example of such a method, a preview image acquired in real time through a camera sensor is displayed in a corresponding application, and a range of a predetermined fixed area to be recognized is displayed in a rectangular or square user interface.

As another example, there may be a method of adjusting the size and position of a specific area to be recognized through a touch screen interface in a captured photo image. In general, since this method is used in a dedicated application for recognizing business cards, barcodes and QR codes, it is often present separately from general camera shooting applications. However, in this case, a user must directly drive a separate application, and since menus or icons exist separately, there may be inconvenience in that the user must find and execute the application.

Alternatively, it can be implemented in the form of adding a separate menu or icon to select the corresponding recognition module inside the camera app. Even in this case, since several sub-menu paths must be passed to select a corresponding menu or icon, the user may feel uncomfortable.

In addition, it occurs when a plurality of objects to be recognized exist or a plurality of types of objects exist in the acquired image. In this case, it is inconvenient to designate a recognition area in an image captured by adjusting a camera pose or zoom for an object to be recognized by the user. In particular, in the case of recognizing a plurality of objects to be recognized consisting of a plurality of recognition module types, it is necessary to sequentially drive a plurality of recognition modules or designate the type of recognition module through a user interface to process them. have.

Various embodiments of the present disclosure may provide a method and apparatus for recognizing a specific object within an electronic device in order to solve the above-described problem.

Various embodiments of the present disclosure may provide a method and method for analyzing an image acquired by an image sensor to acquire information in addition to a basic function of an image sensor in an electronic device, and executing an object recognition module suitable for information.

Various embodiments of the present disclosure determine an area and type of an object to be recognized through a user gesture using a user finger, a stylus pen, or a pointing device when an image acquired through an image sensor is displayed on an electronic device, and recognize the object. By doing so, it is possible to provide a method and apparatus for quickly and conveniently recognizing an object desired by a user.

According to various embodiments of the present disclosure for achieving the above objects, in a method for recognizing a specific object in an image in an electronic device, displaying at least one image, detecting at least one gesture, the According to a gesture, an operation of selecting a recognition function associated with at least one or more objects in the at least one or more images, and an operation of recognizing the at least one or more objects by quoting the selected recognition function.

According to various embodiments of the present invention for achieving the above objects, at least one processor, at least one memory, and at least one program stored in the memory and configured to be executable by the at least one processor. ), wherein the program displays at least one image, detects at least one gesture, selects a recognition function associated with at least one object in the at least one image, according to the gesture, and the By referring to the selected recognition function, commands for recognizing the at least one object may be included.

As described above, by integrating related functions performed by several similar applications that process images acquired from image sensors into one camera application, users can easily access other functions such as contrast, barcode or QR code recognition in addition to the shooting function. You can do it.

In addition, user convenience can be increased by providing a method of defining a specific area and focus of an image through a user gesture through a touch screen in the preview state and selecting a recognition module to analyze image information of the specific area of the image. have.

Various embodiments may be applied not only to barcodes, 2D image codes, face recognition, and character recognition, but also to various fields such as person recognition or object recognition. For example, a service such as character or building recognition may be provided for not only a preview of a camera image or a captured image, but also an image displayed in a state such as playback or pause of a video file.

Various embodiments of the present invention, when there are barcodes, two-dimensional codes, text images, etc. for various products and product names or related information in a print medium such as magazines or newspapers, select a desired code, text, or image, and provide information about this. It may also provide convenience to provide.

Conventional methods require a specific recognition location such as a guide at the center of the screen, and it was difficult to designate or select a specific image or recognition module type on the screen, and in some cases, a method of alternately driving two or more recognition modules is used. This can be a lot.

Various embodiments of the present invention can provide a convenient and fast recognition method because the type of recognition module to be driven is designated or limited according to a user gesture or a selection of a designated area related thereto and a pattern discrimination.

1 illustrates a user interface for selecting a recognition area from an image according to various embodiments of the present disclosure.
FIG. 2 is a diagram illustrating a user interface for driving a recognition module for analyzing image contents in a selected recognition area according to various embodiments of the present disclosure.
3 is a diagram illustrating a user interface for driving a recognition module for analyzing image contents in the selected recognition area by selecting a recognition area in a preview state according to various embodiments of the present disclosure.
4 is a diagram illustrating an example in which information corresponding to the QR code is provided by recognizing a QR code in light and dark according to various embodiments of the present disclosure.
5 is a flowchart for recognizing image content in an electronic device according to various embodiments of the present disclosure.
6 is a flowchart for recognizing image content in an electronic device according to various embodiments of the present disclosure.
7 illustrates an example of recognizing a specific URL (Uniform Resource Locator) address in a news article and entering a site corresponding to the URL address through a web browser according to various embodiments of the present disclosure.
8 illustrates an example of recognizing a specific character in an image path and displaying a plurality of applications related to the recognition result according to various embodiments of the present disclosure.
9 illustrates an example of recognizing a barcode or a QR code according to various embodiments of the present disclosure and displaying a plurality of applications related to the recognition result.
10 is a flowchart illustrating an example of executing an application by recognizing image content in an electronic device according to various embodiments of the present disclosure.
11 illustrates a user interface for selecting a plurality of objects in an image and driving a corresponding recognition module according to various embodiments of the present disclosure.
12 is a flowchart illustrating a flowchart for selecting a plurality of objects in an image and driving a corresponding recognition module according to various embodiments of the present disclosure.
13 is a flowchart for recognizing image content in a preview state of a camera according to various embodiments of the present disclosure.
14 illustrates types of codes including information according to various embodiments of the present invention.
15 illustrates an example of recognizing a QR code based on a user gesture according to various embodiments of the present disclosure.
16 illustrates an example of recognizing a data matrix based on a user gesture according to various embodiments of the present disclosure.
17 illustrates an example of recognizing a PDF 417 or a barcode based on a user gesture according to various embodiments of the present disclosure.
18 is a block diagram of an electronic device according to various embodiments of the present disclosure.
19 is a block diagram of hardware according to various embodiments of the present disclosure.
20 is a block diagram of a programming module according to various embodiments of the present disclosure.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings. Further, in describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary depending on the intention or custom of users or operators. Therefore, the definition should be made based on the contents throughout this specification.

Hereinafter, various embodiments of the present disclosure will be described with respect to a method and an apparatus for recognizing image content in an electronic device.

1 illustrates a user interface for selecting a recognition area from an image according to various embodiments of the present disclosure.

Referring to FIG. 1, a barcode area (a) or a QR code area (b) in the image may be touched and selected by a user. For example, an object closest to the touched position or an object including at least a portion of the touched position may be selected. In various embodiments, the rectangular box 110 may be activated and displayed around the selected object, and the size of the rectangular box may be further adjusted by the user. The rectangular box 110 indicating that an object has been selected is an example, and is not limited to a rectangular box in the present various embodiments, and may be displayed in various shapes. The image may be a preview screen acquired through an image sensor or an image (or video, photo) previously stored in a memory.

In various embodiments, an object region may be selected through input such as hovering, dragging, or sweeping instead of a touch or tap input.

In another embodiment not shown, in the case of an HMD (Head Mounted Display) equipped with a camera or a see-through type eyeglass, an image input through an imaging device is displayed through a display mounted in front of the eye. Therefore, the selection of an object may be determined using a gesture or movement of a hand or finger, not a touch device. To this end, a gesture of a finger or a hand may be detected by the above or a separate camera or a gyro sensor or an acceleration sensor mounted on the hand, and a corresponding object region may be selected according to the gesture. For example, when a gesture of a finger positioned between a corresponding object and an HMD occurs on a specific object, the finger gesture and position may be determined through a camera, and an area of a specific object existing at the corresponding position may be recognized. When a finger points to an object and then a circle gesture is made or the finger is shaken up and down to display, the area of the object at least partially located in the corresponding area can be designated. Alternatively, when a specific object is pointed with a fingertip and held for a predetermined period of time or longer, an operation of recognizing an object positioned in the direction of the fingertip, adjacent to the finger, or at least partially overlapping with the fingertip and selecting a region of the object may be performed. At this time, the specific object and the finger do not necessarily have to physically contact the object, and the closest object image or finger image pointed to by the fingertip image may contact or cover at least a partial area of the object image to determine what the object is. have. However, when the object image is covered, the input image prior to the hiding is stored in advance, and the corresponding object image is restored in memory to check the area of the object image. As another example, when a corresponding gesture occurs, it is recognized that it is an object selection operation, and when the gesture ends and an image of a finger or hand deviates from the object area, the object area may be determined.

When such an object area is selected, the user may be notified that the corresponding object area is selected through graphic, sound, voice, and haptic feedback.

2 is a diagram illustrating a user interface for driving a recognition module for analyzing an object in a selected recognition region according to various embodiments of the present disclosure. Here, the recognition module is a module for recognizing objects such as barcodes, QR codes, and characters, and may be implemented as software or hardware. Hereinafter, it will be referred to as an object recognition module.

Referring to FIG. 2, an example of driving an object recognition module for analyzing a barcode area or a QR code area selected in FIG. 1 is shown.

For example, in Fig. 1(a), when a barcode area is selected by a user touch input and a user gesture (hereinafter referred to as a swipe operation) of dragging from left to right after touching is detected, the object recognition module ( Alternatively, a barcode recognition module) may be driven to recognize a barcode.

In this case, a user gesture such as a swipe operation is not suggested in the various embodiments as an example, and may be variously defined by the user through environment setting.

In addition, a plurality of user gestures and a plurality of object recognition modules are mapped, so that an object recognition module corresponding to the user gesture may be executed immediately.

In other various embodiments, the user gesture is used as a command for driving an object recognition module, and which object recognition module is used may be driven after automatically analyzing an object type and selecting a corresponding object recognition module. Reference will be made to an example of determining an object type in FIGS. 15 to 17 below.

In general, in the case of barcode, business card, 2D code recognition, etc., recognition performance may be affected by the sharpness of the image. Therefore, a better recognition rate can be expected by recognizing user gestures by focusing on the selected object area. . To this end, when an object area is selected, focus is simultaneously focused on the selected object, and object recognition may be performed. In various embodiments, when focus is automatically or manually focused on a specific area, an input area for user gestures is created in a certain area in the focused area, and for recognizing a barcode in an area including at least a portion of the area. When a predetermined user gesture is input, a barcode or a QR code may be recognized and a predetermined related function may be performed.

FIG. 3 is a diagram illustrating a user interface for driving a recognition module for analyzing image contents in the selected recognition area by selecting a recognition area in a preview state according to various embodiments of the present disclosure.

Referring to FIG. 3A, a QR code area may be selected by a user's touch in an image acquired from an image sensor. At this time, the focus may be focused on the QR code area.

According to various embodiments, a QR code may be automatically selected and focused from an image acquired from an image sensor. In other words, the object area including the QR code may be set in the preview state without a user touch input.

Referring to FIG. 3B, a recognition module for recognizing a QR code may be driven according to a user gesture in which a touch is dragged in an L shape in a selected QR code area in the image. That is, it implies that the L-shaped user gesture needs to be driven by the QR code recognition module.

Here, since the QR code recognition module is mapped to the user gesture, the QR code recognition module may be immediately driven when a user gesture is input.

According to various embodiments of the present disclosure, the type of an object in the selected object area may be determined, and an object recognition module determined by a user gesture may be driven.

4 is a diagram illustrating an example in which information corresponding to the QR code is provided by recognizing a QR code in light and dark according to various embodiments of the present disclosure.

Referring to FIG. 4A, a business card is previewed on a display through an image sensor, and as described in FIGS. 1 to 3, an object area is designated and a QR code can be recognized.

Referring to (b) of FIG. 4, when the QR code on the business card is recognized in (a) of FIG. 4, information included in the QR code may be extracted and provided to the display. For example, information such as name, company name, contact information, email address, address, and home page may be displayed.

5 is a flowchart for recognizing image content in an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 5, the electronic device may display an image through a display in operation 500. The image may be an image obtained through an image sensor and previewed and provided, or may be a photo or video previously stored in a memory.

In operation 502, the electronic device may detect at least one user gesture.

* For example, a partial area in the image may be designated as an object area through a first user gesture, and an object recognition module for analyzing the designated object area may be selected through a second user gesture. For example, the first user gesture is a touch adjacent to the object area or detected within the object area, and the second user gesture is mapped to a corresponding object recognition module and may be defined by a user touch pattern input.

In various embodiments, the object area may be automatically designated without the first user gesture. For example, the object area may be automatically designated based on the shape, size, position, color, pattern density, etc. of the object.

In various embodiments, the second user gesture may simply be used as a command instructing to drive an object recognition module, and in this case, the electronic device may determine the type of the object in the designated object area based on a predetermined reference.

In various embodiments, an object area designation and an object recognition module may be determined through one user gesture. For example, when a user gesture is input in a rectangular shape centered on the object area, the size of the rectangle becomes the object area, and the rectangle may indicate that the first object recognition module is used. If a user gesture is input in the shape of a circle around the object area, the size of the circle becomes the object area, and the circle may indicate that the second object recognition module is used.

In various embodiments, the electronic device may additionally perform an operation of improving an image based on the designated object area. For example, when the image is a preview image, a new image with improved image quality may be obtained by focusing based on the designated object area. Alternatively, image quality may be improved based on the brightness, sharpness, contrast, or exposure of the image.

The electronic device may select an object recognition module in operation 504 according to at least one user gesture, analyze a designated object area in the image using the object recognition module selected in operation 506, and display the analysis result in operation 508. .

In various embodiments, the electronic device may display at least one recommended object recognition module module before analyzing a designated object area in the image.

In still other various embodiments, in a wearable display device such as an HMD or eyeglass, the above embodiments may be implemented by recognizing a user's input by a camera or a gesture sensing device mounted on a hand or finger.

6 is a flowchart for recognizing image content in an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 6, the electronic device may display an image through a display in operation 600. The image may be an image obtained through an image sensor and previewed and provided, or may be a photo or video previously stored in a memory.

In operation 602, the electronic device may designate at least one object area in the image.

For example, in the image, a boundary or a range of the object area may be automatically designated according to a predetermined criterion. Here, the predetermined criterion may be based on one or more of a size, a position, and a pattern density of one or more objects arranged in the image. In addition, the predetermined criterion may be focused on the object area by recognizing that the object in the foreground closest to the imaging device or the largest object is recognized.

If a plurality of object regions are recognized in an image, the focus may be sequentially processed at predetermined time intervals.

In various embodiments, the object area in the image may be designated by a user input (or a first user gesture). For example, the object region may be designated as an object closest to the touched position or an object region including at least the touched position.

In various embodiments, the electronic device may additionally perform an operation of improving an image based on the designated object area. For example, when the image is a preview image, a new image with improved image quality may be obtained by focusing based on the designated object area. Alternatively, image quality may be improved based on brightness, sharpness, contrast, or exposure of the image.

In operation 604, the electronic device may detect a second user gesture input signal within the selected object area of the image and display at least one related object recognition module module.

For example, by detecting a user gesture pattern (eg, a swipe operation, an L-type touch input, etc.), a list of at least one object recognition module mapped to the detected user gesture pattern may be displayed.

To this end, the electronic device calculates the pattern information of the input user gesture and the pre-stored user gesture pattern information and a matching rate, and one or more object recognition modules (e.g., character recognition module, barcode recognition) related to one or more candidate patterns having a matching rate of a certain or more. Module, face recognition module).

In other various embodiments, the object recognition module may be determined according to the object type by determining the type of the object area according to the second user gesture input signal.

In operation 606, the electronic device may select one of at least one or more recognition modules according to a user input.

The electronic device may analyze the object region of the image by using the object recognition module selected in operation 608 and display the analysis result in operation 610.

In various embodiments, when there are a plurality of detected object regions, the same operations (operations 602 to 610) may be respectively performed on the objects of the corresponding object regions by one second user gesture.

In the various embodiments, the second user gesture may be detected outside the detected object area. For example, if there is one or more detected object areas, and all other object areas are barcodes, if a user gesture input is a barcode recognition command in an area other than the detected object area, the detected object areas are A decoding command for the barcodes is performed, and the result can be displayed on the display.

In still other various embodiments, without an operation of displaying a list of at least one related object recognition module to select one of the list of object recognition modules, the related object recognition module is immediately driven according to the second user gesture and the detected object area This can be analyzed.

6, an object region is designated by a first user gesture, and an object recognition module to recognize the object region is selected and driven by a second user gesture, but object region designation and object recognition module selection are performed through one user gesture. It can also be done. That is, when a user gesture input signal is input, the object area is designated based on the location of the image where the user gesture input signal is generated, the type of the object area is determined, and the object recognition module corresponding to the result is selected and driven. have.

Still another variety of embodiments may provide various user interfaces related to a designated object recognition module. That is, when an object designated by a user gesture is recognized, or when a pattern of a user gesture is recognized, various services may be provided in relation to the analysis result.

For example, as shown in FIG. 7 below, when there is an advertisement on a page such as a newspaper or a magazine, if there is URL (Uniform Resource Locator) address information in the advertisement content, it may be displayed in various ways. The URL may be displayed in simple text, or in the case of a complex URL, it may be provided in the form of a QR code or barcode. When an object displayed as a text, a barcode, a two-dimensional image code, etc. is acquired through an image sensor and an object recognition module is designated through a user gesture, it is possible to provide various related applications instead of simply recognizing the object.

For example, when the object is decoded and the information is in the form of a URL, if it is determined that information such as a browser, a phone number, and an address is included, a related application such as a phone application or a mail writing application is started, or the user interface is displayed. You can also choose by. One or more types of information may be included in the designated or recognized object, and one or more related applications may be designated or run.

In general, after the object recognition module is driven to recognize information, the related application may be driven or displayed on the display in the form of a user interface so that the related application can be selected, but the object recognition module may not necessarily be executed. Even when the object recognition module is designated by a user gesture, one or more related applications may be provided without performing the object recognition module. For example, the URL information corresponds to text information, and may be used when accessing a site through a web browser, but may be simply copied to memory and then used as information for pasting into another application. Accordingly, when it is determined that the object to be recognized is a character, the operation of driving the character recognition module may be performed after executing a related application or displaying an application list.

In various embodiments, after analyzing the barcode or QR code through the barcode or QR code recognition module as shown in FIG. 8 below, related applications may be displayed.

7 illustrates an example of recognizing a specific URL address in a news article and entering a site corresponding to the URL address through a web browser according to various embodiments of the present disclosure.

Referring to FIG. 7A, based on the method for recognizing the image content of FIG. 5 or 6, a character object (eg, URL address information) is designated in an image, and a character recognition module (eg: It shows an example in which the content of the object area is analyzed through the character recognition module after the Optical Character Reader (OCR) is selected.

Referring to (b) of FIG. 7, an output corresponding to the analysis result of (a) of FIG. 7 is shown. For example, when the "www.samsung.com" URL address is recognized in (a) of FIG. 7, a site corresponding to www.samsung.com may be accessed through a web browser.

8 illustrates an example of recognizing a specific character in an image and displaying a plurality of applications related to the recognition result according to various embodiments of the present disclosure.

Referring to FIG. 8, when previewing through image sensors of a plurality of movie posters, after an object area is designated and a character recognition module is selected, a related application list 820 may be displayed based on the analysis result of the character recognition module. have. In this case, the user can run any one application in the application list.

For example, when a user gesture for character recognition is defined as an operation of drawing a circle on the touch screen, the user may draw a circle 810 in the object area (the word'New York') in the preview state. In this case, after detecting the user gesture applied to the touch panel, the electronic device may recognize that the user gesture is a gesture to extract a character from an image, and cut the image of the object area 810 designated as the user gesture (CROP). ) Text ('New York') can be extracted from the cropped image. In general, when a subject is photographed with an image sensor, a process of focusing may be included. When a specific user gesture is input in a specific object area, the electronic device may use the focus area as a point where the user gesture is input. In this case, the width of the object area according to the user gesture may vary according to the user gesture, and text information may be extracted from the object area and provided to the user when the focus of the subject is correct even if the photographic shutter key is not applied. In addition, when a character is extracted from the object area, a list of applications that can process the character may appear. In various embodiments, when a specific application is selected, an application capable of processing a corresponding character may be executed. For example, when the character “New York” is designated and the focus is focused, the character recognition module (eg OCR) operates, and when a memo application is selected from the application list, the word “New York” can be saved through the memo application. .

In various embodiments, when the content of the image is processed, the camera app may return to the normal shooting mode.

9 illustrates an example of recognizing a barcode or a QR code according to various embodiments of the present disclosure and displaying a plurality of applications related to the recognition result.

Referring to FIG. 9A, an example of recognizing a barcode or a QR code in a specific area of an image in a preview state is shown. In (a) of FIG. 9, a barcode or a QR code may be selected.

Referring to (b) of FIG. 9, an example of displaying a list of related applications on the barcode recognized in (a) of FIG. 9 is shown. For example, when the recognized barcode or information included in the QR code includes email, phone number, homepage information, etc., related applications (e.g., barcode, internet, message, email, memo, phone application) are displayed. Can be.

Referring to (c) of FIG. 9, an example of displaying a list of related applications on the QR code recognized in (a) of FIG. 9 is shown. For example, Internet and phone applications corresponding to the recognized QR code may be displayed.

10 is a flowchart illustrating an example of executing an application by recognizing image content in an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 10, the electronic device may display an image through a display in operation 1000. The image may be an image obtained through an image sensor and previewed and provided, or may be a photo or video previously stored in a memory.

In operation 1002, the electronic device may detect at least one user gesture. For example, a partial area in the image may be designated as an object area through a first user gesture, and an object recognition module for analyzing the designated object area may be selected through a second user gesture. For example, the first user gesture is a touch adjacent to the object area or detected within the object area, and the second user gesture is mapped to a corresponding object recognition module and may be defined by a user touch pattern input.

In various embodiments, the second user gesture may be used as a command for instructing driving the object recognition module, and in this case, the electronic device may determine the type of the object region based on a predetermined criterion. For example, as shown in FIGS. 15 to 17 below, a code type may be determined using the characteristics of the code.

In various embodiments, an object area designation and an object recognition module may be determined through one user gesture. For example, when a user gesture is input in a rectangular shape centered on the object area, the size of the rectangle becomes the object area, and the rectangle may indicate that the first object recognition module is used. If a user gesture is input in the shape of a circle around the object area, the size of the circle becomes the object area, and the circle may indicate that the second object recognition module is used.

In various embodiments, the electronic device may additionally perform an operation of improving an image based on the designated object area. For example, when the image is a preview image, a new image with improved image quality may be obtained by focusing based on the designated object area. Alternatively, image quality may be improved based on the brightness, sharpness, contrast, or exposure of the image.

In operation 1004, the electronic device selects an object recognition module according to at least one user gesture, analyzes at least some object areas in the image using the object recognition module selected in operation 1006, and relates it based on the analysis result in step 1008. Applications can be displayed.

When one of the related applications is selected in operation 1010, the electronic device may proceed to operation 1012 and execute the selected related application.

11 illustrates a user interface for selecting a plurality of objects in an image and driving a corresponding recognition module according to various embodiments of the present disclosure.

Referring to FIG. 11, when different types of objects (business card 1102, product 1 (1000), product 2 (1004)) are positioned in one image, the same user gesture, that is, the left side of the object area An object recognition module corresponding to each object may be driven by dragging a touch from top to bottom right. By limiting the range of the object area for recognizing each object in the image area, it is possible to reduce the amount of computation and memory usage for image processing.

Here, the object recognition module corresponding to each object region may be determined through image processing or pattern analysis.

For example, as shown in FIG. 11, when the user gesture is a diagonal line, the start point of the diagonal line may be the upper left corner of the object area to be designated, and the end point of the diagonal line may be recognized as the lower right side of the object area. As still another various embodiments, a user gesture (eg, a square shape or a circle shape) for including a corresponding object in an object region to be recognized may be used. Alternatively, the user gesture may be a touch, and the touch may be around or inside an object. When the touch exists inside the object, the area including the object can be designated by checking the density of the pattern, the shape of the pattern, the boundary line of the pattern, the excess margin of the pattern, and color distribution.

When designating an object area, it is possible to determine what the type of the corresponding object is by using the pattern distribution constituting the object in the designated area, the shape and frequency of the pattern, color, and brightness information. Alternatively, when a user gesture is input to designate an object recognition module or other function, the recognition module may be designated by examining an image pattern corresponding to the input coordinates. That is, this is a method of using a user gesture input as a scan line for analyzing pixel information of an image. For example, pattern information (e.g., finer pattern, start and end pattern, alignment pattern) included in barcodes or two-dimensional image codes is used as an identification symbol representing information such as the position, direction, and tilt of the code. , Since such a pattern is defined according to a predetermined criterion, the type of the corresponding code can be determined through the pattern information. As shown in FIG. 11, in the case of a barcode pattern when the user's gesture is dragged diagonally, the start pattern and the end pattern, that is, the ratio of the widths of the black bar and the white bar can be recognized, so that it is possible to easily determine the object in the form of a barcode. . Even in the case of the QR code of the business card, the diagonal line of the user's gesture goes through the various patterns of the QR code image. Among the patterns, when passing through the finger pattern, the ratio of black and white, or the difference in brightness is 1:1:3:1:1 Since it is entered as a QR code, it can be easily recognized.

12 is a flowchart illustrating a flowchart for selecting a plurality of objects in an image and driving a corresponding recognition module according to various embodiments of the present disclosure.

Referring to FIG. 12, in operation 1200, the electronic device may display an image through a display. The image may be an image obtained through an image sensor and previewed and provided, or may be a photo or video previously stored in a memory.

The electronic device selects an object region from the image according to a user gesture in operation 1202, determines an object recognition module corresponding to the object region in operation 1204, and identifies the object region in the image using the object recognition module determined in step 1206. Analyze and display the analysis result in operation 1208.

In operation 1210, when another object area is detected in the image by a user gesture, the electronic device may repeatedly perform operations 1202 to 1208.

In various embodiments, after a plurality of object regions are designated, an object recognition module corresponding to the plurality of object regions may be selected and analyzed in parallel.

13 is a flowchart for recognizing image content in a preview state of a camera according to various embodiments of the present disclosure.

Referring to FIG. 13, the electronic device may drive a camera app in operation 1300, display a preview screen in operation 1302, and autofocus in operation 1304.

The electronic device may receive a touch input on the preview screen in operation 1306 and determine whether the touch input is a user gesture in operation 1308.

If the touch input is the user gesture in operation 1308, the user gesture is analyzed in operation 1310, and in operation 1312, a recognition module that matches the user gesture may be selected and focus on the object region.

Here, an object region to be recognized and an object recognition module may be selected through user gesture analysis.

When a region including at least an object is designated through the user gesture, a pattern or a pattern arrangement state of the object is analyzed based on this, and a range necessary for recognizing the object may be recognized. For example, when an object region is automatically designated, an object region to be recognized within an image can be found based on the shape, size, position, color, and pattern density of the object. A typical case is recognizing a face region in a photo or preview image, and it is possible to find facial region candidates mainly using color, and determine whether a face is a face based on location information such as eyebrows, eyes, and mouth in this region. In the case of a barcode or QR code area, after searching for a specific pattern such as a barcode or a two-dimensional image code through a scan line in the image, it is possible to know the location and area of the code using this. In the case of a character area, characters form a cluster of less than a certain distance, and character, number, and symbol area candidates can be recognized using a characteristic that is divided by line. Since there may be more than one such candidate, an operation of extracting several candidates according to a predetermined criterion, for example, size, shape, position, etc., is displayed on the display and selecting one or more of the candidates by a user input device is also added. Can be included as.

In various embodiments, an object area may be designated based on a location designated by a user gesture. For example, when a specific location of an image displayed on the display is designated in the preview image, an operation for designating an area related to the location is performed. To do this, touch or tap, hover, or drag. It is also possible to recognize a location where sweeping has occurred, designate an area of one or more objects that include the location or the nearest one, or designate an object area that includes at least a part of the area designated by the user interface. To this end, it is possible to recognize the size, color, brightness, shape, etc. of the object and determine suitability as a designated area by discriminating the density, color, and boundary of the object using positions designated by the user interface. For example, the surrounding color may be determined based on the designated location, and the same color area may be determined as a face area candidate. In the case of barcodes, there is an extra white space called the Quiet Zone (QZ) around the code pattern. Most of the time, the white space around the barcode is this part, to easily separate the code pattern from the background image. In more detail, since the area is sufficiently larger than the interval between code patterns, it is possible to easily distinguish the code pattern area from the other areas by using the difference in density of the QZ pattern. Even in the case of an alphanumeric region, it is possible to determine the line and text regions by using the density of adjacent objects based on the input location information. In the case of a color code, a plurality of cells may be adjacent to each other to represent a code, such as a rectangle or a circle. Such an object can recognize the shape and boundary of the object by tracking the boundary line.

In various embodiments, in the case of a continuous image such as a preview image, a moving image, and an animated GIF, a user interface indicating the object region may be displayed while maintaining the position of the selected object region. For example, an area recognized as a barcode area may be displayed as translucent, color, brightness, etc., or may be displayed using a border line, mark, or figure, and the user interface may be maintained even in an image newly inputted each time. This is to facilitate providing functions by a user input device, such as selection or gesture input, to a user by continuously tracking a changed position in the case of the candidate object area or the selected object area. Tracking of a changed position can be obtained by calculating a difference image for a previously input image and a subsequent input continuous image (each frame unit or a predetermined time period) to obtain a changed motion vector. Alternatively, each required image may be obtained each time using the above region extraction method. In general, since the method of obtaining each time increases the process time and memory usage, the process of determining whether the object region selection operation needs to be performed, such as when the difference between the images is very large or a new object is input through differential image analysis. It can be done after going through. Or, when a predetermined time elapses from the occurrence time of the previously performed object region selection operation, this is recognized as a condition to be performed in the determination process, and the region designating process is newly inputted to determine the most recent accurate situation. Can be done for

In various embodiments, the code recognition module includes a one-dimensional barcode such as a Universal Product Code (UPS), an International Article Number (EAN), an International Standard Book Number (ISBN), a Quick Response (QR) code, a data matrix, One or more of a variety of image code types, such as a matrix-type two-dimensional code such as Maxi code and ColorCode, and a layered code in the form of multiple layers of barcodes such as PDF-417 May include, and may be selected by a user gesture. Alternatively, when the code recognition module is driven, the code type may be determined and a corresponding code recognition module may be driven.

The operation of designating the object area and the operation of selecting the object recognition module may be operated separately, but may be performed simultaneously in one user gesture. For example, when a user gesture is input, an area including a part of an area in which the user gesture has occurred may be recognized as the object area. The object area may include a location where a touch occurs to start a gesture, a location where the gesture ends, or a location of one or more objects in a path where the gesture occurs.

The location where the gesture occurs may be outside the designated object area. For example, if there is one or more designated object areas and all of the corresponding object areas are barcodes, if a gesture input generated in an area other than the designated object area means a barcode recognition command, decoding of barcodes of the designated object areas The command is executed, and the result can be displayed on the respective display by user selection.

As another embodiment, a type of a decoder or module to be operated may be designated by using pattern information of an image included in a path in which a gesture occurs. For example, in order to recognize a one-dimensional barcode such as EAN or ISBN, when a gesture including an image of the start pattern portion of a barcode pattern is taken, it is easy to recognize it as a one-dimensional barcode. Therefore, if a gesture input occurs in an area including at least one of a start pattern, an end pattern, and a data pattern representing the characteristics of a one-dimensional barcode, the type can be determined by analyzing the pattern information of the image area in which the stroke information occurs when the gesture is input I can. Another example of a similar method may be determined by analyzing unique pattern information (Eye bull mark, code alignment pattern, finder pattern, etc.) of the 2D barcode in the image area through which the gesture passes. For example, in the case of a QR code, since it is a square image code with three finding patterns, the path on the stroke is tracked in the specified object area, and pattern information in the ratio of 1:1:3:1:1, a characteristic of the finder pattern If found, it can be recognized as a QR code.

The electronic device may analyze the object region using the object recognition module selected in step 1314 and provide the analysis result in step 1316.

On the other hand, in step 1308, if the touch input is not the user gesture, the process proceeds to step 1318 to perform a function for camera photographing such as focusing a selected area or zooming in/zooming out.

14 illustrates types of codes including information according to various embodiments of the present disclosure.

14(a) is a data matrix, FIG. 14(b) is a QR code, FIG. 14(c) is PDF417, and FIG. 14(d) is Code 49 .

The data matrix is a high-density two-dimensional barcode capable of encoding up to 3110 characters in a total 256-byte ASCII character set, the QR code is a two-dimensional barcode in a matrix format representing information in a black and white grid pattern, and the PDF417 is a variable It is a two-dimensional symbol code having a symbol length and variable symbol height, and Code 49 is a multi-layered and small data length variable continuous symbology, which can be expressed as all 128 ASCII characters.

15 illustrates an example of recognizing a QR code based on a user gesture according to various embodiments of the present disclosure.

Referring to FIG. 15, the QR code may be composed of a Finder pattern, an alignment pattern, and a data area. The Finder pattern exists in the ratio of a 1:1:3:1:1 black-and-white pattern at each corner of the square, and can be used to detect the location of a QR code in a decoder application. The alignment pattern exists between Finder patterns in a ratio of 1:1:1:1:1 to black and white, and can be used to determine the coordinates of a symbol in a decoder application.

FIG. 15(a) shows a case where a user gesture is detected diagonally from the first Finder pattern to the third Finder pattern, and FIG. 15(b) is from the left to the right from the first Finder pattern to the second Finder pattern. When a user gesture is detected, FIG. 15(c) is a case where a user gesture is detected in a first Finder pattern area, and FIG. 15(d) shows a first Finder pattern-second Finder pattern-third A user gesture is detected in the direction of the Finder pattern, and (e) of FIG. 15 is a case where a user gesture is detected in the direction of the first Finder pattern-the second Finder pattern-the third Finder pattern-the fourth Finder pattern. (F) of 15 is a case in which the character Q in the QR code is input through a user gesture.

As described above, it is possible to recognize the QR code by recognizing the user gesture pattern based on the Finder pattern information of the QR code or through a user gesture of Q character input indicating the QR code.

16 illustrates an example of recognizing a data matrix based on a user gesture according to various embodiments of the present disclosure.

Referring to FIG. 16, like a QR code, a data matrix may be composed of a Finder pattern, an alignment pattern, and a data area. However, the Finder pattern of the data matrix is located at the left boundary and the lower boundary as shown in Fig. 16(b), and a black and white pattern at each of the right boundary and upper boundary as shown in Fig. 16(a) of the data matrix: It can exist in a 1:1:1 ratio.

FIG. 16(a) is a case where a user gesture is detected based on an alignment pattern existing in a ratio of 1:1:1:1:1 to a black-and-white pattern on a right border and an upper border, and FIG. 16(b) ) Is a case where a user gesture is detected based on a Finder pattern existing at the left and lower borders, and FIG. 16C shows a black and white pattern at the right and upper borders, respectively, 1:1:1:1: A user gesture is detected based on the alignment pattern present at a ratio of 1 and the Finder pattern present at the left and lower boundaries, and FIG. 16D shows the character "D" in the data matrix through the user gesture. This is the case when it is entered.

As described above, the data matrix may be recognized by recognizing the user gesture pattern based on the Finder pattern and alignment pattern information of the data matrix, or the data matrix may be recognized through a user gesture of D character input indicating the data matrix.

17 illustrates an example of recognizing a PDF 417 or a barcode based on a user gesture according to various embodiments of the present disclosure.

Referring to FIG. 17, PDF417 may be composed of a start pattern, an end pattern, and a data area. The start pattern signals the beginning of the code, and the end pattern signals the end of the code. Although not shown, there may be a left row indicator indicating that it is left after the start pattern, and a Right Row indicator indicating that it is right next to the end pattern may be located. In addition, PDF417 may include an alignment pattern in the middle as shown in (c) of FIG. 17.

Figure 17 (a) is a case where the user gesture is detected in a straight line from the point where the start pattern is present to the point where the end pattern is, and Figures 17 (b) and (c) show the end pattern at the point where the start pattern is present. A user gesture to a point is detected in a diagonal direction, and FIG. 17D shows a case in which the character “P” is input in the data matrix through the user gesture.

As described above, the PDF417 may recognize the user's gesture pattern based on the start pattern, end pattern, and alignment pattern information to recognize the PDF417 or recognize the PDF417 through a user gesture of the P character input indicating the PDF417.

The electronic device according to the present disclosure may be a device including a communication function. For example, a smart phone (smartphone), a tablet PC (tablet personal computer), a mobile phone (mobile phone), a videophone, an e-book reader (e-book reader), a desktop PC (desktop personal computer), a laptop PC (laptop PC) personal computer), netbook computer, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical device, electronic bracelet, electronic necklace, electronic appcessory, camera, Wearable devices, electronic clocks, wrist watches, smart white appliances (e.g. refrigerators, air conditioners, vacuum cleaners, artificial intelligence robots, TVs, digital video disk (DVD) players) , Audio, oven, microwave oven, washing machine, air purifier, electronic frame, etc.), various medical devices (e.g. MRA (magnetic resonance angiography), MRI (magnetic resonance imaging), CT (computed tomography), camera, ultrasound, etc.), Navigation device, global positioning system receiver (GPS), event data recorder (EDR), flight data recorder (FDR), set-top box, TV box (e.g., Samsung HomeSyncTM, Apple TVTM) , Or Google TVTM), electronic dictionaries, automotive infotainment devices, electronic equipment for ships (e.g., navigation devices for ships, gyro compasses, etc.), avionics, security devices, electronic clothing , Electronic key, cam Coders, game consoles, head-mounted displays (HMD), flat panel display devices, electronic albums, furniture or parts of buildings/structures including communication functions, electronic boards It may be a combination of one or more of various devices such as an electronic board, an electronic signature receiving device, or a projector. . It is obvious to those skilled in the art that the electronic device according to the present disclosure is not limited to the above-described devices.

18 is a block diagram of an electronic device according to various embodiments of the present disclosure. Referring to FIG. 18, the electronic device 1800 may include a bus 1810, a processor 1820, a memory 1830, a user input module 1840, a display module 1850, or a communication module 1860. I can.

* The bus 1810 may be a circuit that connects the above-described components to each other and transmits communication (eg, a control message) between the above-described components.

The processor 1820, for example, through the bus 1810, other components described above (for example, the memory 1830, the user input module 1840, the display module 1850), the communication The module 1860 or the like) may receive an instruction, decrypt the received instruction, and perform an operation or data processing according to the decrypted instruction.

The memory 1830 may be received from the processor 1820 or other components (eg, the user input module 1840, the display module 1850, the communication module 1860, etc.) or the processor 1820 ) Or commands or data generated by other components. The memory 1830 may include programming modules such as, for example, a kernel 1831, middleware 1832, an application programming interface (API) 1833 or an application 1834. Each of the above-described programming modules may be composed of software, firmware, hardware, or a combination of at least two or more of them.

The kernel 1831 is the system resources used to execute the other programming modules, for example, the middleware 1832, the API 1833, or an operation or function implemented in the application 1834 (e.g. : The bus 1810, the processor 1820, the memory 1830, etc.) can be controlled or managed. In addition, the kernel 1831 may provide an interface that allows the middleware 1832, the API 1833, or the application 1834 to access and control or manage individual components of the electronic device 1800. have.

The middleware 1832 may serve as an intermediary so that the API 1833 or the application 1834 communicates with the kernel 1831 to exchange data. In addition, the middleware 1832 is, for example, in relation to the work requests received from the (multiple) applications 1834, to at least one of the (multiple) applications 1834 Load on a work request using a method such as assigning priority to use system resources of the electronic device 1800 (for example, the bus 1810, the processor 1820, or the memory 1830) You can perform balancing.

The API 1833 is an interface that allows the application 1834 to control functions provided by the kernel 1831 or the middleware 1832, for example, file control, window control, image processing, or text control. It may include at least one interface or function for the like.

The user input module 1840 may receive, for example, a command or data from a user and transmit it to the processor 1820 or the memory 1830 through the bus 1810. The display module 1850 may display an image, an image, or data to a user.

The communication module 1860 may connect communication between another electronic device 1802 and the electronic device 1800. The communication module 1860 includes a predetermined short-range communication protocol (e.g., Wifi (wireless fidelity), BT (Bluetooth), NFC (near field communication)) or a predetermined network communication (e.g. Internet, LAN (local area network), WAN). (wire area network), telecommunication network, cellular network, satellite network, or POTS (plain old telephone service), etc.) 1862. Each of the electronic devices 102 and 104 is the same as the electronic device 1800 (eg It may be a device of the same type or a different device (eg, a different type).

In addition, the communication module 1860 may be connected to another electronic device 1804 and a server 1864 through a network 1862 between the electronic device 1800.

19 is a block diagram of hardware 1900 according to various embodiments. The hardware 1900 may be, for example, the electronic device 1800 illustrated in FIG. 18. Referring to FIG. 19, the hardware 1900 includes one or more processors 1910, a subscriber identification module (SIM) card 1914, a memory 1920, a communication module 1930, a sensor module 1940, and a user input module. (1950), a display module (1960), an interface 270, an audio codec (1980), a camera module (1991), a power management module (1995), a battery (1996), an indicator (1997) or a motor (1998). can do.

The processor 1910 (for example, the processor 1820) may include one or more application processors (AP) 1911 or one or more communication processors (CP) 1913. The processor 1910 may be, for example, the processor 1820 shown in FIG. 18. 19 shows that the AP 1911 and the CP 1913 are included in the processor 1910, the AP 1911 and the CP 1913 may be included in different IC packages, respectively. In one embodiment, the AP 1911 and the CP 1913 may be included in one IC package.

The AP 1911 may drive an operating system or an application program to control a plurality of hardware or software components connected to the AP 1911, and perform various data processing and operations including multimedia data. The AP 1911 may be implemented with, for example, a system on chip (SoC). According to an embodiment, the processor 1910 may further include a Graphic Processing Unit (GPU) (not shown).

The CP 1913 manages a data link and converts a communication protocol in communication between an electronic device (eg, the electronic device 1800) including the hardware 1900 and other electronic devices connected through a network. can do. The CP 1913 may be implemented as an SoC, for example. According to an embodiment, the CP 1913 may perform at least a part of a multimedia control function. The CP 1913 may distinguish and authenticate a terminal within a communication network using, for example, a subscriber identification module (eg, a SIM card 1914). In addition, the CP 1913 may provide services such as a voice call, a video call, a text message, or packet data to the user.

In addition, the CP 1913 may control data transmission/reception of the communication module 1930. In FIG. 19, components such as the CP 1913, the power management module 1995, or the memory 1920 are shown as separate components from the AP 1911, but according to an embodiment, the The AP 1911 may be implemented to include at least some of the aforementioned components (eg, the CP 1913).

According to an embodiment, the AP 1911 or the CP 1913 may load commands or data received from at least one of a nonvolatile memory or other components connected to each of the volatile memory to be processed. . Further, the AP 1911 or the CP 1913 may store data received from at least one of the other components or generated by at least one of the other components in a nonvolatile memory.

The SIM card 1914 may be a card implementing a subscriber identification module, and may be inserted into a slot formed at a specific location of an electronic device. The SIM card 1914 may include unique identification information (eg, integrated circuit card identifier (ICCID)) or subscriber information (eg, international mobile subscriber identity (IMSI)).

The memory 1920 may include an internal memory 1922 or an external memory 1924. The memory 1920 may be, for example, the memory 1830 shown in FIG. 18. The internal memory 1922 may be, for example, a volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.)) or a non-volatile memory, for example. For example, OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, NAND flash memory, NOR flash memory, etc.) It may include at least one. According to an embodiment, the internal memory 222 may take the form of a solid state drive (SSD). The external memory 1924 is, for example, compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), or a memorystick. It may further include.

The communication module 1930 may include a wireless communication module 1931 or an RF module 1934. The communication module 1930 may be, for example, the communication module 1860 shown in FIG. 18. The wireless communication module 1931 may include, for example, WiFi 233, BT (bluetooth) 1935, GPS 1937, or NFC (near field communication) 1939. For example, the wireless communication module 1931 may provide a wireless communication function using a radio frequency. Additionally or alternatively, the wireless communication module 1931 connects the hardware 1900 to a network (eg, Internet, a local area network (LAN), a wire area network (WAN), a telecommunication network, a cellular network, a satellite network, or a POTS). (plain old telephone service), etc.) may include a network interface (eg, LAN card) or a modem to connect.

The RF module 1934 may be responsible for transmitting and receiving data, for example, transmitting and receiving an RF signal or a called electronic signal. Although not shown, the RF module 1934 may include, for example, a transceiver, a power amp module (PAM), a frequency filter, or a low noise amplifier (LNA). In addition, the RF module 1934 may further include a component for transmitting and receiving an electromagnetic wave in a free space in wireless communication, for example, a conductor or a conducting wire.

The sensor module 1940 is, for example, a gesture sensor 1940A, a gyro sensor 1940B, an atmospheric pressure sensor 1940C, a magnetic sensor 1940D, an acceleration sensor 1940E, a grip sensor 1940F, and a proximity sensor. (1940G), RGB (red, green, blue) sensor (1940H), biometric sensor (1940I), temperature/humidity sensor (1940J), illuminance sensor (1940K), or UV (ultra violet) sensor (1940M) Can include. The sensor module 1940 may measure a physical quantity or detect an operating state of an electronic device, and convert the measured or detected information into an electric signal. Additionally/alternatively, the sensor module 1940 may include, for example, an olfactory sensor (E-nose sensor, not shown), an EMG sensor (electromyography sensor, not shown), an EEG sensor (electroencephalogram sensor, not shown), and ECG. A sensor (electrocardiogram sensor, not shown) or a fingerprint sensor may be included. The sensor module 1940 may further include a control circuit for controlling at least one or more sensors included therein.

The user input module 1950 may include a touch panel 1952, a (digital) pen sensor 1954, a key 1956, or an ultrasonic input device 1958. . The user input module 1950 may be, for example, the user input module 1840 shown in FIG. 18. The touch panel 1 952 may recognize a touch input using at least one of, for example, a capacitive type, a pressure sensitive type, an infrared type, or an ultrasonic type. In addition, the touch panel 1952 may further include a controller (not shown). In the case of capacitive type, not only direct touch but also proximity recognition is possible. The touch panel 1952 may further include a tactile layer. In this case, the touch panel 1952 may provide a tactile reaction to a user.

The (digital) pen sensor 1954 may be implemented using, for example, a method identical or similar to that of receiving a user's touch input or a separate recognition sheet. As the key 1956, a keypad or a touch key may be used, for example. The ultrasonic input device 1958 is a device capable of checking data by detecting sound waves with a microphone (eg, a microphone 1988) in a terminal through a pen generating an ultrasonic signal, and wireless recognition is possible. According to an embodiment, the hardware 1900 may use the communication module 1930 to receive a user input from an external device (eg, a network, a computer, or a server) connected thereto.

The display module 1960 may include a panel 1962 or a hologram 264. The display module 1960 may be, for example, the display module 1950 illustrated in FIG. 18. The panel 1962 may be, for example, a liquid-crystal display (LCD) or an active-matrix organic light-emitting diode (AM-OLED). The panel 1962 may be implemented, for example, to be flexible, transparent, or wearable. The panel 262 may be configured with the touch panel 1952 as a single module. The hologram 1964 may show a three-dimensional image in the air using interference of light. According to an embodiment, the display module 1960 may further include a control circuit for controlling the panel 1962 or the hologram 1964.

The interface 1970 includes, for example, a high-definition multimedia interface (HDMI) 1972, a universal serial bus (USB) 1974, a projector 1976, or a D-sub miniature (1978). Can include. Additionally or alternatively, the interface 1970 may include, for example, a secure digital (SD)/multi-media card (MMC) (not shown) or an infrared data association (IrDA) (not shown).

The audio codec 1980 may convert voice and electric signals bidirectionally. The audio codec 1980 may convert voice information input or output through, for example, a speaker 1982, a receiver 1984, an earphone 1986, or a microphone 1988.

The camera module 1991 is a device capable of capturing images and video, and according to an embodiment, one or more image sensors (eg, front lens or rear lens), image signal processor (ISP), or flash LED ( Flash LED, not shown) may be included.

The power management module 1995 may manage power of the hardware 1900. Although not shown, the power management module 1995 may include, for example, a power management integrated circuit (PMIC), a charger integrated circuit (IC), or a battery fuel gauge.

The PMIC may be mounted in an integrated circuit or an SoC semiconductor, for example. Charging methods can be divided into wired and wireless. The charger IC may charge a battery and prevent overvoltage or overcurrent from flowing from a charger. According to an embodiment, the charger IC may include a charger IC for at least one of a wired charging method or a wireless charging method. The wireless charging method includes, for example, a magnetic resonance method, a magnetic induction method, or an electromagnetic wave method, and additional circuits for wireless charging, such as a coil loop, a resonance circuit, and a rectifier, may be added. have.

The battery gauge may measure, for example, the remaining amount of the battery 1996 and a voltage, current, or temperature during charging. The battery 1996 may generate electricity to supply power, and may be, for example, a rechargeable battery.

The indicator 1997 may display a specific state of the hardware 1900 or a part thereof (for example, the AP 1911), for example, a boot state, a message state, or a charging state. The motor 1998 may convert an electrical signal into mechanical vibration. The MCU 1999 may control the sensor module 1940.

Although not shown, the hardware 1900 may include a processing device (eg, a GPU) for supporting mobile TV. The processing device for supporting mobile TV may process media data according to standards such as digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or media flow.

Names of the above-described components of the hardware according to the present disclosure may vary according to the type of electronic device. The hardware according to the present disclosure may be configured to include at least one of the above-described components, and some components may be omitted or additional other components may be further included. In addition, some of the components of the hardware according to the present disclosure are combined to form a single entity, so that functions of the corresponding components before the combination can be performed in the same manner.

20 is a block diagram of a programming module 2000 according to an embodiment. The programming module 2000 may be included (eg, stored) in the electronic device 1800 (eg, the memory 1830) illustrated in FIG. 18. At least a part of the programming module 2000 may be composed of software, firmware, hardware, or a combination of at least two or more of them. The programming module 2000 is implemented in hardware (e.g., the hardware 1900) and is an operating system (OS) that controls resources related to an electronic device (e.g., the electronic device 1800) or is driven on the operating system. Various applications (eg, the application 2070) may be included. For example, the operating system is Android, iOS, Windows, Symbian, Tizen, or Bada. 20, the programming module 2000 may include a kernel 2010, middleware 2030, an application programming interface (API) 2060, or an application 2070.

The kernel 2010 (for example, the kernel 131) may include a system resource manager 2011 or a device driver 312. The system resource manager 2011 may include, for example, a process management unit 2013, a memory management unit 2015, or a file system management unit 2017. The system resource manager 2011 may control, allocate, or retrieve system resources. The device driver 2012 may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WiFi driver, or an audio driver. In addition, according to an embodiment, the device driver 2012 may include an IPC (inter-process communication, not shown) driver.

The middleware 2030 may include a plurality of modules previously implemented in order to provide functions commonly required by the application 2070. In addition, the middleware 2030 may provide a function through the API 2060 so that the application 2070 can efficiently use limited system resources inside the electronic device. For example, as shown in FIG. 20, the middleware 2030 (eg, the middleware 1832) is a runtime library 2035, an image recognition manager 3136, an application manager 2041, and a window manager. (window manager) (2042), multimedia manager (2043), resource manager (2044), power manager (2045), database manager (2046), package manager (2047), connectivity manager (2048), notification manager (2049), location manager (2050), graphic manager (2051) or security manager It may include at least one of (2052).

The runtime library 2035 may include, for example, a library module used by a compiler to add a new function through a programming language while the application 2070 is being executed. According to an embodiment, the runtime library 2035 may perform input/output, memory management, or functions for arithmetic functions.

The application manager 2041 may manage a life cycle of at least one application among the applications 2070, for example. The window manager 2042 may manage GUI resources used on a screen. The multimedia manager 2043 may grasp a format required for reproduction of various media files, and may perform encoding or decoding of a media file using a codec suitable for a corresponding format. The resource manager 2044 may manage resources such as source code, memory, or storage space of at least one application among the applications 2070.

The power manager 2045 may operate together with a basic input/output system (BIOS) to manage a battery or power and provide power information necessary for operation. The database manager 2046 may manage to create, search, or change a database to be used by at least one application among the applications 2070. The package manager 2047 may manage installation or update of an application distributed in the form of a package file.

The connection manager 2048 may manage a wireless connection such as WiFi or Bluetooth. The notification manager 2049 may display or notify an event such as an arrival message, an appointment, and a proximity notification in a manner that does not interfere with the user. The location manager 2050 may manage location information of an electronic device. The graphic manager 2051 may manage a graphic effect to be provided to a user or a user interface related thereto. The security manager 2052 may provide all security functions required for system security or user authentication. According to an embodiment, when an electronic device (for example, the electronic device 1800) has a phone function, the middleware 2030 is a call manager for managing the voice or video call function of the electronic device. manager, not shown).

The image recognition manager 2136 may manage a user interface related to selection of an object area and selection of an object recognition module in an image of the present disclosure.

According to an embodiment, objects or areas such as people, objects, faces, backgrounds, etc. are extracted and separated from a photographed image, and key attributes corresponding to each object or each area (object position in the image, relative position between objects/areas) Or, it is a module that identifies one or more of the phase, shape, color, speaker-related information, metadata, etc.). By transferring and storing the main attributes of the identified object or region to memory, when an information query occurs for the object or region included in the image, the process can receive and process one or more attribute information corresponding thereto from the memory. have. The information query can be image, audio, coordinates or area designation by user input interface, query by keyword, etc.

The image recognition manager 2136 may manage various image processing such as pre-processing, image segmentation, feature extraction, recognition, and recognition.

For example, the image processing techniques described below may be used.

By using attribute conversion techniques such as binarization, gray-level scaling, and color conversion, unnecessary information can be removed or processing speed can be accelerated. For example, to extract the shape of a specific object from an image, rather than using a color image, using a binarized black and white image can gain a gain in terms of processing speed. For example, characters, numbers, one-dimensional barcodes, two-dimensional monochrome barcodes, and face regions may be first converted into black-and-white through binarization, thereby reducing the amount of computation when calculating pattern information or boundary line extraction.

When using digital image transformation techniques such as Fourier Transformation, Discrete Cosine Transformation, and Wavelet Transformation, it is easy to apply techniques such as high-pass filtering and low-pass filtering. Image enhancement and region of interest extraction can be processed.

Image enhancement techniques such as blurring, sharpening, expansion, reduction, noise reduction, smoothing, and brightness adjustment are performed by analyzing the information of the pixels constituting an image. By using it, it can be used for feature extraction and recognition by reducing or restoring the loss of major attributes such as the shape, position, and color of the object in the image.

The process of determining the size of a digital image, the size of a specific area, and the shape of an object through edge detection, boundary following, feature point extraction, silhouette extraction, color separation, area coloring algorithm, etc. I need this. In addition, it can be used for image extraction, classification, and recognition by finding the outline of a digital image, distinguishing colors and patterns, and finding textures.

For motion and gesture recognition, two or more images are analyzed to obtain a difference image between the two images, and through this, the movement pattern or amount of change of the object is investigated to determine movement, motion, gesture, etc.

The image recognition manager 2136 may extract an image area and set an image area for user input. By applying one or more of the above image processing techniques, an object region such as a face region, a silhouette of a person or object, and a motion (motion) of the object may be extracted. In addition, information such as a range, coordinates, border data, and silhouette data of the corresponding region can be managed to be stored in a memory.

The image recognition manager 2136 may display a recognition result processed using an augmented reality, virtual reality (VR), and mixed reality (MR) technique. For example, in the case of a 2D image code, pitch, yaw, and roll information of a camera or object can be obtained according to a finder pattern, an alignment pattern, or a distribution of the code pattern.

For example, in the case of a QR code, the finder pattern exists in the upper left, upper right, and lower left of the four corners of the code image, and does not exist at the lower right, so even if the code image is rotated, the rotated angle of the code can be easily determined. As another example, in the case of a data matrix, the left and right sides are thick solid lines among the boundary portions of the code image and are composed of a 1:1 monochrome pattern, so that the rotation angle of the code can be determined. In the case of a 1-D barcode or PDF-417, direction detection is possible by using parity information according to the start pattern, the shape of the end pattern, and the scan direction of the pattern.

In addition, since the vertical normal vector of the code image can be estimated according to the shape of the code image placed on the (x, y) plane, it is easy to calculate the position and shape of a virtual object in augmented reality or virtual reality in 3D space. I can. Accordingly, 3D or 2D virtual objects (GUI information such as list menus, photos, avatars, and 3D characters) may be displayed by matching according to the pattern of the image using the pose information.

The middleware 2030 may create and use a new middleware module through a combination of various functions of the aforementioned internal component modules. The middleware 2030 may provide modules specialized for each type of operating system in order to provide differentiated functions. In addition, the middleware 2030 may dynamically delete some of the existing components or add new components. Accordingly, some of the components described in the embodiments of the present disclosure may be omitted, other components may be further provided, or components having different names performing similar functions may be substituted.

The API 360 (for example, the API 1833) is a set of API programming functions, and may be provided in different configurations according to an operating system. For example, in the case of Android or iOS, for example, one API set may be provided for each platform, and in the case of Tizen, for example, two or more API sets may be provided.

The application 2070 (for example, the application 1834) may include, for example, a preloaded application or a third party application. For example, the application 2070 is home (2071), dialer (2072), SMS/MMS (2073), IM (2074), browser (2075), camera (2076), alarm (2077), contact ( 2078), voice dial 2079, email 2080, calendar 2081, media player 2082, album 2083, clock 2084, and the like.

At least a part of the programming module 2000 may be implemented as a command stored in a computer-readable storage media. When the command is executed by one or more processors (eg, the processor 1910), the one or more processors may perform a function corresponding to the command. The computer-readable storage medium may be, for example, the memory 1920. At least a part of the programming module 2000 may be implemented (for example, executed) by the processor 1910. At least a portion of the programming module 2000 may include, for example, a module, a program, a routine, a set of instructions or a process for performing one or more functions.

Names of components of a programming module (eg, the programming module 2000) according to the present disclosure may vary depending on the type of operating system. In addition, the programming module according to the present disclosure may include at least one or more of the above-described components, some of the above-described components may be omitted, or additional other components may be further included.

The methods according to the embodiments described in the claims and/or the specification of the present invention may be implemented in the form of hardware, software, or a combination of hardware and software.

When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. One or more programs stored in a computer-readable storage medium are configured to be executable by one or more processors in an electronic device (device). The one or more programs include instructions that cause the electronic device to execute methods according to the embodiments described in the claims and/or specification of the present invention.

These programs (software modules, software) include random access memory, non-volatile memory including flash memory, ROM (Read Only Memory), and electrically erasable programmable ROM. (EEPROM, Electrically Erasable Programmable Read Only Memory), magnetic disc storage device, Compact Disc-ROM (CD-ROM), Digital Versatile Discs (DVDs), or other types of It may be stored in an optical storage device or a magnetic cassette. Alternatively, it may be stored in a memory composed of a combination of some or all of them. In addition, a plurality of configuration memories may be included.

In addition, through a communication network composed of a communication network such as the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a combination thereof It may be stored in an attachable storage device that can be accessed. Such a storage device can access the electronic device through an external port.

In addition, a separate storage device on the communication network may access the portable electronic device.

Meanwhile, although specific embodiments have been described in the detailed description of the present invention, various modifications may be made without departing from the scope of the present invention. Therefore, the scope of the present invention is limited to the described embodiments and should not be determined, and should be determined by the scope of the claims and equivalents as well as the scope of the claims to be described later.

1800: electronics,
1801: bus,
1820: processor,
1830: memory,
1840: user input module,
1850: display module,
1860: communication module.

Claims (30)

In a portable communication device,
display; And
Including a processor, wherein the processor
A photographic image including an object including first information including a first information type and second information corresponding to a second information type is displayed through the display,
Receiving a user input related to the displayed object through the display,
Based at least in part on a determination that the user input has been received in relation to the first information, displaying a first recognition result generated using a first recognition function corresponding to the first information type through the display, and
A portable for displaying a second recognition result generated using a second recognition function corresponding to the second information type through the display based at least in part on a determination that the user input has been received in relation to the second information Communication device.
The method of claim 1,
The processor is
Identifying text as at least part of the first information type, and
A portable communication device for identifying a non-text object as at least part of the second information type.
The method of claim 1,
The processor is
As at least a part of the first recognition function, a portable communication device configured to display the first recognition result generated using an optical character reader (OCR) function.
The method of claim 1,
The processor is
As at least part of the second recognition function, a portable communication device configured to display the second recognition result generated using an image recognition function.
The method of claim 1,
The second information includes a quick response (QR) code,
The processor is
As at least a part of the second recognition function, a portable communication device configured to display the second recognition result generated using a code recognition function.
The method of claim 1,
The processor is
A portable communication device configured to display a menu representing one or more functions related to a corresponding information type among the first information type and the second information type together with a corresponding recognition result among the first recognition result and the second recognition result.
The method of claim 6,
The processor is
A portable communication device configured to display an icon representing a corresponding function among the one or more menus as at least part of the menu.
The method of claim 1,
The display includes a touch screen display,
The processor is
A portable communication device configured to determine corresponding information among the first information and the second information from the displayed object based at least in part on a position on the touch screen display where the user input is received.
The method of claim 1,
The user input includes a user gesture,
The processor is
A portable communication device configured to select corresponding information from among the first information and the second information from the displayed object based at least in part on the user gesture.
The method of claim 1,
A portable communication device wherein a corresponding recognition function among the first recognition function and the second recognition function is selected from a plurality of recognition functions.
The method of claim 10,
Further comprising a memory for storing the plurality of recognition functions,
The processor is
A portable communication device configured to select a corresponding recognition function from among the first recognition function and the second recognition function from the plurality of recognition functions stored in the memory.
In a portable communication device,
display; And
Comprising a processor, the processor
Displaying a photographic image including an object through the display,
Based at least in part on a determination that the displayed object includes first information corresponding to a first information type, a first recognition result generated using a first recognition function corresponding to the first information type is displayed through the display. Mark, and
Based at least in part on a determination that the displayed object includes second information corresponding to a second information type, a second recognition result generated by using a second recognition function corresponding to the second information type is displayed through the display. A portable communication device to display.
The method of claim 12,
The processor is
Identifying text as at least part of the first information type, and
A portable communication device for identifying a non-text object as at least part of the second information type.
The method of claim 12,
The processor is
As at least a part of the first recognition function, a portable communication device configured to display the first recognition result generated using an optical character reader (OCR) function.
The method of claim 12,
The processor is
As at least part of the second recognition function, a portable communication device configured to display the second recognition result generated using an image recognition function.
The method of claim 12,
The second information includes a quick response (QR) code,
The processor is
As at least a part of the second recognition function, a portable communication device configured to display the second recognition result generated using a code recognition function.
The method of claim 12,
The processor is
A portable communication device configured to display a menu representing one or more functions related to a corresponding information type among the first information type and the second information type together with a corresponding recognition result among the first recognition result and the second recognition result.
The method of claim 12,
The display includes a touch screen display,
The processor is
Receiving a user input related to the displayed object through the display,
A portable communication device configured to determine the displayed object based at least in part on a position on the touch screen display where the user input is received in response to the reception of the user input.
A non-transitory computer-readable recording medium storing instructions for causing the at least one processor to perform set operations when executed by at least one processor, wherein the operations are
Displaying a photographic image including an object including first information including a first information type and second information corresponding to a second information type, and a display operatively connected to the at least one processor Displaying through;
Receiving a user input related to the displayed object through the display;
Displaying, through the display, a first recognition result generated using a first recognition function corresponding to the first information type based at least in part on a determination that the user input has been received in relation to the first information; And
Displaying, through the display, a second recognition result generated using a second recognition function corresponding to the second information type based at least in part on a determination that the user input has been received in relation to the second information. A recording medium containing.
The method of claim 19,
The above actions are
A recording medium further comprising displaying a menu representing one or more functions related to a corresponding information type among the first information type and the second information type together with a corresponding recognition result among the first recognition result and the second recognition result. .
In a portable communication device,
camera;
display;
Comprising a processor, the processor
Control the camera to capture an image,
A captured image including an object including a first image part and a second image part is displayed through the display,
Receiving a user input related to the displayed object through the display,
Based at least in part on a determination that the user input has been received in relation to the first image portion, displaying, through the display, a first set of user interface items for executing functions associated with the first image portion, and
Based at least in part on a determination that the user input has been received in relation to the second image portion, displaying, through the display, a second set of user interface items for executing functions related to the second image portion,
The first set of user interface items is displayed corresponding to that a first type of image recognition function is applicable to the first image portion,
The second set of user interface items is displayed in response to a second type of image recognition function applicable to the second image portion.
The method of claim 21,
The first type of image recognition function corresponds to an object recognition function,
The second type of image recognition function corresponds to a text recognition function.
The method of claim 21,
The first image portion corresponds to first information, the second image portion corresponds to second information,
The first recognition result is generated based on the first information, and the second recognition result is generated based on the second information.
The method of claim 23,
A first item corresponding to the first recognition result is displayed together with the first set of user interface items, and a second item corresponding to the second recognition result is displayed together with the second set of user interface items Portable communication device.
The method of claim 24,
The at least one first item further includes an icon representing a function using the first information.
The method of claim 21,
A portable communication device wherein a non-text object is identified from the first image portion through the first type of image recognition function, and one or more characters are identified from the second image portion through the second type of image recognition function.
The method of claim 26,
The processor is
Controlling the display to display a first result obtained based on the identified non-text object, and
A portable communication device that controls the display to display a second result obtained based on the identified one or more characters.
The method of claim 21,
The second image portion includes a quick response (QR) code,
The processor is
A portable communication device that controls the display to display a third result obtained through a code recognition function based on the QR code.
The method of claim 21,
Further comprising a memory for storing a plurality of recognition functions,
The processor is
A portable communication device configured to select a corresponding image recognition function from among the first type of image recognition function and the second type of image recognition function from the plurality of recognition functions stored in the memory.
The method of claim 21,
The display includes a touch screen display,
The processor is
A portable communication device configured to determine a corresponding image portion of the first image portion and the second image portion from the displayed object based at least in part on a position on the touch screen display where the user input is received.

Images