KR20230000932A - Methods and devices for analyzing images - Google Patents

Abstract

The present invention relates to a method and device for analyzing an image. According to one embodiment of the present invention, a method for detecting an object may include the steps of: extracting frame images and motion vectors from images, moving pictures, image information, and/or moving picture information; generating an integrated feature vector based on the frame image and the motion vector; and detecting an object included in the images, the moving pictures, the image information, and/or the moving picture information based on the integrated feature vector. It is possible to provide the method and device for effectively detecting an object included in the images, the moving pictures, the image information, and/or the moving picture information.

Description

Methods and devices for analyzing images {Methods and devices for analyzing images}

The present invention relates to a method for extracting an object from an image and an apparatus therefor.

In addition, the present invention also relates to a method and apparatus for extracting an object from content and providing information on the extracted object. Specifically, the present invention relates to a method and an apparatus for providing information on a specific object to a user when the user desires information on a specific object in content.

Recently, thanks to the rapid increase in multimedia contents and the generalization of security equipment such as cctv, the importance of image recognition technology for moving images such as broadcasting contents and cctv recorded images is increasing.

In general, video-based image recognition technology includes a technology based on a still image and a technology based on a plurality of consecutive frame images. A technology based on still images divides a video into still images in frame units, and detects and recognizes objects by applying image-based analysis technology to each still image. A technique based on a plurality of consecutive frame images detects a moving object or recognizes a specific event by modeling motion characteristics of an object based on a plurality of frame images.

However, there is a limit to high-speed recognition of time-series motion due to the complexity and excessive amount of computation of a plurality of state models or a plurality of consecutive frame image models. Furthermore, image recognition technology in the field of security using CCTV provides a technology for separating and recognizing a moving object from a video with a fixed background, but it is difficult to detect a specific object or separate an object from a moving camera image or dynamic background. There are limits.

In addition, with the recent development of information and communication technology, users are often exposed to content such as photos and videos. At this time, users want to obtain information on a specific object on the content, but information on the object is not provided on the content. In addition, there may be cases in which users want to obtain information on a specific object not only in content but also in daily life. Specifically, there are cases in which a user desires information on a specific object (eg, electronic product, clothes, bag, etc.) encountered on media or in daily life.

However, in order to obtain information on a specific object, the user has to search the Internet or the like, and there are cases in which information on a specific product cannot be obtained because a user cannot accurately input a keyword or the like for Internet search.

Accordingly, a need for a method for accurately providing information on a specific object desired by a user and an apparatus therefor has emerged.

Domestic Patent Publication 10-2015-0005131 A (2015.01.14) Domestic Patent Publication 10-2014-0122292 A (2014.10.20)

An embodiment of the present invention effectively detects an image, video, image information, and/or an object included in video information based on an integrated feature vector, taking into account both static and dynamic characteristics of the object and effectively , a method and apparatus for detecting an object included in image information and/or video information may be provided.

In addition, an embodiment of the present invention detects an object included in an image, a video, image information, and/or video information based on an integrated feature vector, effectively taking into account both static and dynamic characteristics of the object, A method and apparatus for detecting a video, image information, and/or an object included in the video information may be provided.

In addition, an embodiment of the present invention is a method and apparatus for detecting an object at high speed by effectively reducing the amount of computation by combining the advantages of simplicity and calculation efficiency of object detection based on still images and high performance of object detection based on continuous multiple frame images. can provide.

In addition, an embodiment of the present invention is a method for detecting an object showing a consistent movement pattern with high accuracy by integrating and utilizing image information of an object included in a still image and motion information of an object such as total/partial movement and transformation, and device can be provided.

In addition, an embodiment of the present invention considers both the static characteristics of an object based on a frame image and the dynamic characteristics of an object based on a motion vector, so that an object may be displayed in a photographed image, video, image information, and/or video information. It is possible to provide an object detection method that is robust against possible blurring.

In addition, an embodiment of the present invention may provide an apparatus for providing information on a specific object within content.

In addition, an embodiment of the present invention may provide a method of operating a server providing information on a specific object within content.

An embodiment of the present invention proposes a method and apparatus for detecting an object based on a frame image and a motion vector.

An object detection method according to an embodiment of the present invention includes extracting frame images and motion vectors from images, videos, image information, and/or video information; generating an integrated feature vector based on the frame image and the motion vector; and detecting an object included in the image, video, image information, and/or video information based on the integrated feature vector.

In the object detection method according to an embodiment of the present invention, the generating of the integrated feature vector may include extracting statistical characteristics of the frame image as a first feature vector and converting statistical characteristics of the motion vector into a second feature vector. Extracting with; and generating the integrated feature vector by combining the first feature vector and the second feature vector.

The step of extracting the first feature vector and the second feature vector in the object detection method according to an embodiment of the present invention includes dividing the frame image and the motion vector into a plurality of blocks, and the frame image included in the block. A first feature vector may be extracted based on , and a second feature vector may be extracted based on a motion vector included in a block.

In the object detection method according to an embodiment of the present invention, the extracting of the first feature vector and the second feature vector may include extracting the first feature vector based on a brightness gradient of a pixel included in the frame image. can

In the object detection method according to an embodiment of the present invention, the extracting of the first feature vector and the second feature vector may include extracting the first feature vector based on a brightness level of a pixel included in the frame image. there is.

In the object detection method according to an embodiment of the present invention, the extracting of the first feature vector and the second feature vector may include extracting the first feature vector based on a color of a pixel included in the frame image. .

In the step of extracting the first feature vector and the second feature vector in the object detection method according to an embodiment of the present invention, the second feature vector may be extracted based on the direction of the motion vector.

In the object detection method according to an embodiment of the present invention, the step of extracting the frame image and the motion vector may include dividing a reference frame corresponding to the frame image into a plurality of blocks and extracting a motion vector for each block to map the motion vector. , and the size of the blocks constituting the motion vector map may be equalized.

In the object detection method according to an embodiment of the present invention, the step of detecting an object included in the image, video, image information, and/or video information may include a detection target object in the frame image based on the integrated feature vector. It is possible to detect an object included in the image, video, image information, and/or video information by determining whether it is included.

In the object detection method according to an embodiment of the present invention, the step of extracting the frame image and the motion vector may include extracting the image, video, image information, and/or motion vector included in the video information in a decoding process, or A motion vector may be extracted based on an image, a video, image information, and/or a plurality of consecutive frame images included in the video information.

An object detection apparatus according to an embodiment of the present invention includes an extractor for extracting frame images and motion vectors from images, videos, image information, and/or video information; a feature generating unit generating an integrated feature vector based on the frame image and the motion vector; and an object detector detecting an object included in the image, video, image information, and/or video information based on the integrated feature vector.

In the object detection apparatus according to an embodiment of the present invention, the feature generation unit extracts statistical characteristics of the frame image as a first feature vector, extracts statistical characteristics of the motion vector as a second feature vector, and The integrated feature vector may be generated by combining the first feature vector and the second feature vector.

In the object detection apparatus according to an embodiment of the present invention, the feature generator divides the frame image and the motion vector into a plurality of blocks, extracts a first feature vector based on the frame image included in the block, A second feature vector may be extracted based on the motion vector included in the block.

In the object detection apparatus according to an embodiment of the present invention, the feature generator may extract the first feature vector based on a gradient of brightness of pixels included in the frame image.

In the object detection apparatus according to an embodiment of the present invention, the feature generator may extract the first feature vector based on a brightness level of a pixel included in the frame image.

In the object detection apparatus according to an embodiment of the present invention, the feature generator may extract the first feature vector based on a color of a pixel included in the frame image.

In the object detection apparatus according to an embodiment of the present invention, the feature generator may extract the second feature vector based on the direction of the motion vector.

The step of detecting an object included in the image, video, image information, and/or video information in the object detection apparatus according to an embodiment of the present invention may include a detection target object in the frame image based on the integrated feature vector. It is possible to detect an object included in the image, video, image information, and/or video information by determining whether it is included.

In addition, an embodiment of the present invention provides a user device for obtaining information on a specific object in content.

A user device according to an embodiment of the present invention includes a communication unit; and a processor functionally connected to the communication unit, wherein the processor obtains content including one or more objects, extracts a specific object from among the one or more objects of the content, and selects the specific object through a deep neural network. Alternatively, identification information for identifying the specific content extracted based on pre-learned learning information using a convolutional neural network is transmitted to an object information providing server, and from the object information providing server, one or more information about the specific object is transmitted. Information is received and displayed, one or more information on the specific object is different according to the type of the specific object, and the one or more information on the specific object is overlapped with the content or displayed in a pop-up form. do.

The specific object may be extracted based on pixels having pixel values of pixels adjacent to each of the pixels constituting the content equal to or greater than a predetermined threshold value.

The predetermined threshold value may be an average value of pixel values of pixels constituting the content.

The one or more pieces of information on the specific object include at least one of a person's name, date of birth, an item worn by the person, and a place where the item was purchased when the specific object is a person, and a model name when the particular object is an object. , the date of manufacture, the price, and at least one of the place of purchase of the product.

The number of one or more pieces of information about the specific object may be determined by the amount of money paid by the user to the object information providing server.

In addition, an embodiment of the present invention is a method for acquiring information on a specific object in content by a device, comprising: acquiring content including one or more objects; extracting a specific object from among the one or more objects of the content, extracting the specific object based on pre-learned learning information using a deep neural network or a convolutional neural network; Transmitting identification information for identifying the specific content to an object information providing server; and receiving one or more pieces of information about the specific object and displaying the information from the object information providing server through a display. Including, the one or more information on the specific object is different according to the type of the specific object, and the one or more information on the specific object overlaps with the content or is displayed in a pop-up form.

In addition, an embodiment of the present invention proposes a computer readable recording medium recording a program for executing the method.

In addition, an embodiment of the present invention proposes a program recorded on a computer readable recording medium to execute the method.

An embodiment of the present invention effectively detects an image, a video, image information, and/or an object included in the video information based on an integrated feature vector, taking into account both static and dynamic characteristics of the object. A video, image information, and/or an object included in the video information may be detected.

An embodiment of the present invention can detect an object at high speed by effectively reducing the amount of computation by combining the simplicity and calculation efficiency of still image-based object detection and the high performance of object detection based on a plurality of consecutive frame images.

An embodiment of the present invention can detect an object exhibiting a consistent movement pattern with high accuracy by integrating and utilizing image information of an object included in a still image and movement information of an object such as total/partial movement and transformation. .

An embodiment of the present invention considers both the static characteristics of an object based on a frame image and the dynamic characteristics of an object based on a motion vector, thereby appearing in an image, moving image, image information, and/or moving image information of an object captured. It is possible to provide an object detection method that is robust against possible blurring.

There is an effect that the user can be immediately provided with information on a specific object in the content desired by the user.

The effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

Also other aspects as described above, features and benefits of certain preferred embodiments of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings.
1 is a diagram illustrating an object detection method according to an embodiment of the present invention.
2 is a diagram for explaining a process of generating an integrated feature vector according to an embodiment of the present invention.
3 is a diagram illustrating an example of generating an integrated feature vector from images, videos, image information, and/or video information according to an embodiment of the present invention.
4 is a diagram showing a detailed configuration of an object detection device according to an embodiment of the present invention.
5 is a diagram showing the configuration of a system for providing information on a specific object in content according to an embodiment of the present invention.
6 is a block diagram illustrating a user device according to an embodiment of the present invention.
7 is a diagram illustrating a method of providing information on a specific object within content according to an embodiment of the present invention.
8 is a diagram illustrating a method of providing information on a specific object within content according to an embodiment of the present invention.
9 is a flowchart illustrating a method of providing information on a specific object within content according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In describing the embodiments, descriptions of technical contents that are well known in the technical field to which the present invention pertains and are not directly related to the present invention will be omitted. This is to more clearly convey the gist of the present invention without obscuring it by omitting unnecessary description.

For the same reason, in the accompanying drawings, some components are exaggerated, omitted, or schematically illustrated. Also, the size of each component does not entirely reflect the actual size. In each figure, the same reference number is assigned to the same or corresponding component.

Advantages and features of the present invention, and methods for achieving them, will become clear with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs It is provided to fully inform the holder of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numbers designate like elements throughout the specification.

At this time, it will be understood that each block of the process flow chart diagrams and combinations of the flow chart diagrams can be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, so that the instructions executed by the processor of the computer or other programmable data processing equipment are described in the flowchart block(s). It creates means to perform functions. These computer program instructions may also be stored in a computer usable or computer readable memory that can be directed to a computer or other programmable data processing equipment to implement functionality in a particular way, such that the computer usable or computer readable memory The instructions stored in are also capable of producing an article of manufacture containing instruction means that perform the functions described in the flowchart block(s). The computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to generate computer or other programmable data processing equipment. Instructions for performing processing equipment may also provide steps for performing the functions described in the flowchart block(s).

Additionally, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). It should also be noted that in some alternative implementations it is possible for the functions mentioned in the blocks to occur out of order. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in reverse order depending on their function.

At this time, the term '~unit' used in this embodiment means software or a hardware component such as a field-programmable gate array (FPGA) or application specific integrated circuit (ASIC), and what role does '~unit' have? perform them However, '~ part' is not limited to software or hardware. '~bu' may be configured to be in an addressable storage medium and may be configured to reproduce one or more processors. Therefore, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

In describing the embodiments of the present invention in detail, an example of a specific system will be the main target, but the main subject matter to be claimed in this specification extends the scope disclosed herein to other communication systems and services having a similar technical background. It can be applied within a range that does not deviate greatly, and this will be possible with the judgment of those skilled in the art.

In the present invention, an object detection device, an object information providing server, and/or a user device may be an application (mobile app) and/or web that implements an online platform that provides services (and/or functions) according to an embodiment of the present invention. Detecting an object from an image, video, image information, and/or video information included in information obtained (and/or received) through a site (web service) and/or providing information about the detected object to the user I'm trying to suggest a feature. Meanwhile, commands (commands) for implementing these functions may be implemented in the form of an extension program in the Chrome browser, a smartphone application, and/or software for augmented reality equipment such as Oculus.

In addition, in the present invention, an object detection device, an object information providing server, and/or a user device may include images and/or images acquired by a camera (and/or sensor) installed inside and/or outside a vehicle according to an embodiment of the present invention. Alternatively, a feature of detecting an object from a video and/or providing information on the detected object to a user is proposed.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an object detection method according to an embodiment of the present invention.

An object detection method according to an embodiment of the present invention may be performed by a processor (and/or control unit) included in the object detection apparatus 400 to be described later. For example, the object detection device 400 is a device for detecting an object included in an image and/or video, and may be implemented as a software module, a hardware module, or a combination thereof.

Also, the object detection device 400 may be implemented as the object information providing server 520 and/or the user device 530 of FIG. 5 . That is, the object detection device 400 may be installed in various computing devices and/or systems such as smart phones, tablet computers, laptop computers, desktop computers, televisions, wearable devices, security systems, and smart home systems. In addition, the object detection device 400 may implement features according to an embodiment of the present invention described with reference to FIGS. 1 to 9 by the object information providing server 520 and the user device 530, respectively, or provide object information. It may also be implemented by the server 520 and the user device 530 together.

In step S110, the object detection apparatus 400 extracts frame images and motion vectors from images and/or moving pictures. The static image may include, for example, at least one of a picture and a photograph having extensions such as jpg and PNG. The static image may include, for example, at least one of a dynamic picture and a photograph having an extension such as GIF. A video may be configured in various forms such as a stream having an extension such as avi or mp4, a file, or a broadcast signal.

The object detection device 400 extracts a frame image from a video. The object detection apparatus may extract a specific frame image by extracting a plurality of frame images included in a video.

The object detection device 400 extracts a motion vector from a video. For example, the object detection device may extract a motion vector included in a video in a process of decoding the video. The motion vector included in the video may be generated during encoding of the video.

As another example, the object detection device 400 may extract a motion vector from a video using a motion vector calculation algorithm. Specifically, the object detection apparatus 400 may calculate an optical flow from a plurality of consecutive frame images extracted from a video. The object detection apparatus 400 may extract a motion vector based on the calculated optical flow. In this case, the object detection apparatus 400 may generate a motion vector map by dividing the reference frame into a plurality of blocks and extracting a motion vector for each corresponding block. The reference frame corresponds to an image frame and may indicate a frame from which a motion vector is extracted.

The size of a plurality of blocks constituting the motion vector map may not be constant. In this case, the object detection apparatus 400 may adjust the size of the plurality of blocks constituting the motion vector map to the smallest block size among the sizes of the plurality of blocks. The object detection apparatus 400 may uniformize the size of blocks constituting the motion vector map.

In step S120, the object detection apparatus 400 generates an integrated feature vector based on the frame image and the motion vector. The object detection apparatus 400 may extract a first feature vector from the frame image and extract a second feature vector based on the motion vector. The object detection apparatus 400 may generate an integrated feature vector based on the first feature vector and the second feature vector.

For example, the object detection apparatus 400 divides a frame image and a motion vector into a plurality of blocks, extracts a first feature vector from the frame image included in the block, and extracts a second feature vector from the motion vector included in the block. can be extracted. The object detection apparatus may generate an integrated feature vector corresponding to the corresponding block by combining the first feature vector and the second feature vector extracted from the corresponding block.

A detailed process of generating the integrated feature vector will be described later with reference to FIG. 2 .

In step S130, the object detection device 400 detects an object included in the image and/or video based on the integrated feature vector. The object detection apparatus may detect an object included in an image and/or a video by determining whether or not the object to be detected is included in the frame image based on the integrated feature vector. The detection target object may represent a moving object included in an image and/or a video. The object to be detected may be included in a partial region of the frame image, and may be included in one block or a plurality of blocks among divided blocks.

The object detection device 400 may operate in different object detection modes according to a predetermined criterion, and set (and/or operate) one of the first object detection mode, the second object detection mode, and the third object detection mode. It can be. For example, the object detection apparatus 400 may generate image information and/or video information by combining a predetermined identifier with an image and/or video received from the outside and/or stored in the object detection apparatus 400. At this time, the predetermined identifier is information indicating the time when the image and / or video was created (and / or captured), the size of the image and / or video, and the detection target object included in the image and / or video It may be information indicating the number, information indicating the number of branches of red-green-blue (RGB) values included in the image and/or video, and the like.

At this time, the information indicating the number of objects to be detected included in the image and/or video is provided by the user directly to the device (and/or terminal, camera, etc.) that captured (and/or created) the image and/or video. It is input information or is first included in the image and / or video by using a predetermined object extraction algorithm in the device (and / or terminal, camera, etc.) that took (and / or created) the image and / or video It may be information that is input (and/or generated) after identifying only the number of objects in advance.

At this time, the predetermined object extraction algorithm is logistic regression, SVM (Support Vector Machine), Latent SVM (Latent Support Vector Machine), deformable part model, HOG (Histogram of Oriented Gradient), Haar- It can include like feature, co-occurrence HOG, local binary pattern (LBP), features from accelerated segment test (FAST), etc.

In addition, information indicating the number of branches of red-green-blue (RGB) values included in the image and/or video (which may be referred to as 'RGB branch number information' for short), for example, When the RGB value of the image and/or video identified (and/or confirmed) by the object detection device 400 (and/or processor) in the image and/or video is FFFFCC, FFCC99, FFCC33, FFCCFF, FF66FF , the number of branches of RGB values included in the image and/or video will be '5'. As another example, when the RGB values of the image and/or video are 33FFFF, 33CCCC, 339999, 336633, 0066FF, 777777, 666666, 330000, 330033, 660066, and 6600CC, the RGB values included in the image and/or video The number of branches will be '11'.

Meanwhile, operation of the object detection apparatus 400 in different object detection modes according to a predetermined criterion will be described again below.

For example, when the image information and/or the video information indicates that the detection target object is a single object and the RGB number information indicates a value higher than a predetermined threshold value (eg, an integer value between 0 and 150), the object is detected. The device 400 may operate in the first object detection mode or may be set to the first object detection mode. The object detection apparatus 400 operating in the first object detection mode or set to the first object detection mode uses various recognizers such as logistic regression, support vector machine (SVM), and latent support vector machine (SVM). object information may be generated (and/or obtained) by detecting an object included in an image, video, image information, and/or video information using

As another example, when the image information and/or video information indicates that the detection target object is a single object, and the RGB number information indicates a value less than or equal to the predetermined threshold value (eg, an integer value between 0 and 150), the object The detection device 400 may operate in the second object detection mode or may be set to the second object detection mode. The object detection apparatus 400 operating in the second object detection mode or set to the second object detection mode replaces the image part model with the image-motion hybrid feature-based part model in the deformable part model, thereby performing regular motion. It is possible to separate the moving object from the background by performing modeling on the object that shows the . Thus, the object detection apparatus may generate (and/or obtain) object information by detecting an object that exhibits regular motion, such as a rotating car wheel or a walking person's leg.

As another example, when the image information and/or the moving image information indicates that the object to be detected is a plurality of objects (in this case, comparison between the RGB number information and the predetermined threshold is not considered), the object detection device 400 may be operated in the third object detection mode or set to the third object detection mode. The object detection device 400 operating in the third object detection mode or set to the third object detection mode may obtain object information by extracting an object included in an image, video, image information, and/or video information. The object detection apparatus 400 operating in the third object detection mode or set to the third object detection mode The object detection apparatus 400 has a Histogram of Oriented Gradient (HOG), Haar-like feature, Co-occurrence HOG, and local binary (LBP) pattern), FAST (features from accelerated segment test) through various algorithms for object feature extraction, etc., images, videos, image information, and/or contours of objects included in video information or text that can be extracted from the objects (or an outline (or outline) representing information) can be obtained. In addition, the object detection device 400 operating in the third object detection mode or set to the third object detection mode recognizes (or identifies) an object included in an image, video, image information, and/or video information through image analysis and , Masking image information may be generated by masking a region corresponding to the recognized object. At this time, the masking process is, for example, a difference image method, a model of Gaussian (MOG) algorithm using GMM (Gaussian Mixture Models), a codebook algorithm, etc. Object information may be extracted and/or obtained by using a method of extracting an object candidate region corresponding to .

2 is a diagram for explaining a process of generating an integrated feature vector according to an embodiment of the present invention.

Step S120 performed by the object detection device may be subdivided as follows.

In operation S121, the object detection apparatus may extract a first feature vector from the frame image and a second feature vector from the motion vector. The object detection apparatus may extract statistical characteristics of a frame image as a first feature vector and extract statistical characteristics of a motion vector as a second feature vector.

The object detection apparatus may divide the frame image and the motion vector into a plurality of blocks. The object detection apparatus may generate an integrated feature vector corresponding to the block by extracting the first feature vector and the second feature vector corresponding to the block for each divided block.

For example, the object detection apparatus may extract a first feature vector based on a brightness gradient of a pixel included in a frame image. The object detection apparatus may extract a first feature vector based on a histogram of a brightness gradient of a pixel.

As another example, the object detection apparatus may extract the first feature vector based on the brightness level of pixels included in the frame image. The object detection apparatus may extract the first feature vector based on the histogram of the brightness level of the pixel.

As another example, the object detection apparatus may extract a first feature vector based on a color of a pixel included in a frame image. The object detection apparatus may extract the first feature vector based on the histogram of the color of the pixel.

For example, the object detection apparatus may extract the second feature vector based on the direction of the motion vector. The object detection apparatus may extract the second feature vector based on a histogram of a direction of at least one motion vector corresponding to the divided block. For example, when there are a plurality of motion vectors included in a divided block, the object detection apparatus may extract a second feature vector based on a vector direction derived by summing the motion vectors included in the block.

In operation S122, the object detection apparatus may generate an integrated feature vector by combining the first feature vector and the second feature vector. The integrated feature vector may refer to a feature vector considering both the first feature vector and the second feature vector. By using the integrated feature vector, the object detection apparatus may detect an object by considering both static and dynamic characteristics of an object included in an image, video, image information, and/or video information.

3 is a diagram illustrating an example of generating an integrated feature vector from images, videos, image information, and/or video information according to an embodiment of the present invention.

According to an embodiment of the present invention, a triangular object and a circular object may be included in the image, video, image information, and/or video information shown in FIG. 3 . FIG. 3 assumes a situation in which a triangular object moves downward and a circular object moves to the top left of an image, video, image information, and/or video information. In the image, video, image information, and/or video information shown in FIG. 3, a solid line may indicate an object after a certain time has elapsed from the dotted line.

The object detection device may extract a frame image from a video. The object detection apparatus may extract a specific frame image by extracting a plurality of temporally continuous frame images included in a video. The object detection apparatus may statically analyze an object included in a video based on the extracted frame image.

The object detection device may extract a motion vector from a video. For example, the object detection device may extract a motion vector generated in a video encoding process from a video. As another example, the object detection apparatus may extract a motion vector from a plurality of temporally continuous frame images included in a video. In this case, the object detection device may extract the motion vector using a motion vector calculation algorithm such as optical flow calculation. The object detection apparatus may divide the reference frame into a plurality of blocks and individually extract motion vectors corresponding to the blocks.

For example, the object detection apparatus may extract a motion vector corresponding to a corresponding block based on a color difference of an image corresponding to the corresponding block. The object detection device compares the current image and the previous image corresponding to the corresponding block, and when the color difference between the two images is greater than a predetermined value, identifies a reference object centered on the part where the color difference appears, and determines the motion of the reference object. By calculating the motion vector, the motion vector of the corresponding block can be extracted. The object detection device may construct a motion vector map using the extracted motion vector. When the size of blocks constituting the motion vector map is not constant, the object detection apparatus may uniformize the blocks included in the motion vector map based on the smallest block size.

The object detection apparatus may dynamically analyze an object included in a video based on a motion vector.

The object detection apparatus may extract a first feature vector from the extracted frame image. The object detection apparatus may divide a frame image into a plurality of blocks and extract a first feature vector for a corresponding block based on the frame image corresponding to the block. For example, the first feature vector for a block may be extracted based on a histogram of a brightness gradient of a pixel included in the corresponding block. As another example, the first feature vector for a block may be extracted based on a histogram of brightness levels of pixels included in the corresponding block. As another example, the first feature vector for a block may be extracted based on a histogram of colors of pixels included in the corresponding block.

The object detection device may extract the second feature vector from the extracted motion vector. The object detection apparatus may extract the second feature vector based on a block having the same size as a block of the frame image. The object detection apparatus may extract a second feature vector corresponding to a corresponding block based on a histogram of a direction of at least one motion vector included in a block having the same size as a block dividing a frame image.

The object detection apparatus may generate an integrated feature vector by combining the first feature vector and the second feature vector. In this case, a block corresponding to the first feature vector and a block corresponding to the second feature vector may have the same size. The object detection apparatus may combine a first feature vector and a second feature vector corresponding to a corresponding block on a block-by-block basis. In other words, the object detection apparatus may generate an integrated feature vector for each region.

4 is a diagram showing a detailed configuration of an object detection device according to an embodiment of the present invention.

Referring to FIG. 4 , the object detection apparatus 400 includes an extraction unit 410 , a feature generation unit 420 and an object detection unit 430 . The object detection device 400 represents a device for detecting an object included in an image, video, image information, and/or video information. The object detection device 400 may be implemented as a software module, a hardware module, or a combination thereof. The object detection device 400 may be installed in various computing devices and/or systems such as smart phones, tablet computers, laptop computers, desktop computers, televisions, wearable devices, security systems, and smart home systems.

The extractor 410 may extract frame images and motion vectors from images, videos, image information, and/or video information. The extractor 410 may extract a specific frame image by extracting a plurality of temporally continuous frame images included in the video.

The extractor 410 may extract a motion vector generated in the video encoding process from the video. Alternatively, the extractor 410 may extract a motion vector based on a plurality of temporally continuous frame images included in the video.

For example, although it is illustrated in FIG. 4 that both frame images and motion vectors are extracted by the extractor 410, this is only one embodiment and does not limit the embodiment of the extractor 410. That is, the object detection apparatus 400 may independently include a frame image extractor extracting a frame image from a video and a motion vector extractor extracting a motion vector from the video.

The feature generator 420 generates an integrated feature vector based on the frame image and the motion vector. The feature generator 420 may divide the frame image into a plurality of blocks and extract a first feature vector corresponding to the block based on the frame image included in the block. The feature generator 420 may extract statistical characteristics of the frame image as a first feature vector.

For example, the feature generator 420 may extract a first feature vector corresponding to a block based on a brightness gradient of a pixel included in a frame image corresponding to the block. As another example, the feature generator 420 may extract a first feature vector corresponding to a block based on a brightness level of a pixel included in a frame image corresponding to the block. As another example, the feature generator 420 may extract a first feature vector corresponding to a block based on a color of a pixel included in a frame image corresponding to the block.

The feature generator 420 may divide the motion vector into a plurality of blocks and extract a second feature vector corresponding to the block based on the motion vector included in the block. The feature generator 420 may extract statistical characteristics of motion vectors as second feature vectors. For example, the feature generator 420 may extract a second feature vector based on the direction of at least one motion vector included in the block. Here, a block dividing the motion vector may have the same size as a block dividing the frame image.

The object detection unit 430 detects an object included in an image, video, image information, and/or video information based on the integrated feature vector. The object detector 430 may detect an image, a video, image information, and/or an object included in the video information by determining whether or not the object to be detected is included in the frame image based on the integrated feature vector.

Some of the techniques applicable to the present invention may be omitted to avoid obscuring the concept of the present invention. These omitted configurations can be applied to the present invention by referring to "Histograms of oriented gradients for human detection" and "Object Detection with Discriminatively Trained Part Based Models".

5 is a diagram showing the configuration of a system for providing information on an object within specific content according to an embodiment of the present invention.

Referring to FIG. 5 , a system for providing information on an object within specific content according to the present invention may include an object information providing server 520 and a user device 530 connected through a communication network 510. there is.

Here, the communication network 510 may be a wired/wireless communication network accessible to the object information providing server 520 and the user device 530 . In addition, the communication network 510 means a connection structure capable of exchanging information between nodes such as terminals and servers, and examples of such a communication network 510 include the Internet, Wireless LAN (Wireless Local Area Network) , Wide Area Network (WAN), Personal Area Network (PAN), 3G, Long Term Evolution (LTE), Wireless Fidelity (WiFi), World Interoperability for Microwave Access (WiMAX), Wireless Gigabit (WiGig), etc. It doesn't work.

The object information providing server 520 may perform a function of providing information on a specific object desired by a user in content according to the present invention to the user device 530 through the communication network 510 . In addition, the object information providing server 520 may provide content to the user device 530 and information on objects included in the provided content. The object information providing server 520 may identify an object received from the user device 530 through the communication network 510 and provide information about the object to the user. In this case, the information about the object may be provided from the outside. For example, information about an object may be provided from a portal site.

The user device 530 may perform a function of receiving content from the outside or receiving content from a user and displaying the content according to the present invention. In addition, the user may select a specific object within the content through the user device 530 . The user device 530 may transmit a specific object selected by the user to the object information providing server 520 through the communication network 510 . The user device 530 may receive information on the selected specific object from the object information providing server 520 through the communication network 510, display the received information on the specific object, and provide the information to the user. In this case, information on a specific object provided to the user may vary depending on the specific object selected by the user, and only part of the information on the specific object may be provided according to the user. The user device 530 is specifically, a communicable desktop computer, a laptop computer, a notebook, a smart phone, a tablet PC, and a mobile phone. , smart watch, smart glass, e-book reader, PMP (portable multimedia player), portable game device, navigation device, digital camera, DMB (digital multimedia broadcasting) player , digital audio recorder, digital audio player, digital video recorder, digital video player, personal digital assistant (PDA), head up Display, HUD), etc. In addition, the user device 530 may include a camera for acquiring an image of a specific object for which the user wants to obtain information. That is, the user may acquire an image of a specific object for which information is desired through the user device 530 . The user transmits the obtained image to the object information providing object information providing server 520 connected to the user device 530 through the communication network 510, and the specific object in the image obtained from the object information providing object information providing server 520 information can be provided.

6 is a block diagram illustrating a user device according to an embodiment of the present invention.

Referring to FIG. 6 , a user device 530 according to an embodiment of the present invention includes a display unit 610, a storage unit 620, a communication unit 630, an object extraction unit 640, and an input unit 650. can be configured. All or part of the display unit 610, the storage unit 620, the communication unit 630, the object extraction unit 640, and the input unit 650 constituting the user device in this specification may be described as a processor. For example, the user device may be composed of a communication unit 630 and a processor (display unit 610, storage unit 620, object extraction unit 640, and input unit 650), wherein the processor and the communication unit are functional with each other. can be connected to

The display unit 610 may perform a function of displaying content, receiving information on a specific object included in the content from the object information providing server 520 and displaying the received information. For example, the display unit 610 may display content previously stored by the user, content captured by the user in real time, or content received from the outside in real time. . In this case, the content may be a video, image (photo), audio (voice), or the like.

The storage unit 620 may perform a function of storing various contents previously stored by a user. Also, the storage unit 620 may perform a function of storing information about a specific object included in content selected by a user.

The communication unit 630 may perform communication between the user device 530 and the object information providing server 520 through the communication network 510 described in FIG. 5 . That is, a specific object selected by the user from content can be transmitted to the object information providing server 520, and information on a specific object can be received from the object information providing server 520. In addition, when an Internet link is included in the information on a specific object, a function of allowing the user device 530 to access the Internet link may be performed.

The object extractor 640 may extract an object included in content pre-stored in the storage 620 or content directly photographed by a user. The object extractor 640 may extract an object based on pre-learned information using a deep neural network (DNN) or a convolutional neural network (CNN). In order to display the objects extracted through the object extractor 640, a separate bounding box may be overlapped and displayed on the content. For example, when one or more objects exist in content, a bounding box may be assigned to each object. Also, a separate bounding box may exist in a transparent state without being displayed within the content. If the content is a video or image, a pixel value may be used to extract an object included in the content. That is, the object extractor 640 may extract a specific object based on the similarity between pixels of the content. Specifically, in order to extract an object, the background and object must be separated from the video or image. In many cases, the pixel values of each pixel are similar between pixels corresponding to the background and objects corresponding to the object. . Accordingly, the object extractor 640 may obtain pixel values of all pixels of the video or image, and extract an object based on pixels for which a difference between pixel values of adjacent pixels is equal to or greater than a predetermined threshold value. An adjacent pixel may refer to a pixel above, below, left, or right of the current pixel. In other words, for each pixel in the content, pixel value differences with pixels adjacent to each pixel are calculated, and when the pixel value differences exceed a predetermined threshold value, a set of corresponding pixels may become the aforementioned bounding box. . In this case, the predetermined threshold value may be a value corresponding to an average of pixel values of all pixels of the video or image.

A pixel value described in this specification may be a value representing brightness of each pixel. For example, if the content is black and white, the pixel value may be any one of values from 0 to 255. When the pixel value is 0, it means the darkest state (ie, black), and when the pixel value is 255, it means the brightest state (ie, white).

As another example, when the content is color content rather than black and white, each pixel may be expressed as a mixture of three colors of red (R), green (G), and blue (B). A pixel value of each pixel may include a value corresponding to R, a value corresponding to G, and a value corresponding to B, respectively. Values corresponding to R, G, and B may have any one of values from 0 to 255, respectively. When the value corresponding to R is 0, it means black, and when it is 255, it may mean red itself (ie, the primary color). The same applies to the values corresponding to G and B. The pixel value difference described above may be calculated for each value corresponding to R, G, and B of each pixel. In other words, the difference between pixel values of adjacent pixels may be calculated as a value corresponding to R, a value corresponding to G, and a value corresponding to B, respectively. Also, the average of pixel values of all pixels may be an average of values corresponding to R, an average of values corresponding to G, and an average of values corresponding to B of all pixels. That is, for each of the pixel values corresponding to R, G, and B, a difference value and a predetermined threshold value are compared. Specifically, when differences between pixel values corresponding to R, G, and B are equal to or greater than a predetermined threshold value, a bounding box may be set.

Meanwhile, if any one of the differences between pixel values corresponding to R, G, and B is equal to or greater than a predetermined threshold value, a bounding box may be set. In this case, when a difference between pixel values corresponding to an element having the smallest average value among averages of R, G, and B pixel values of all pixels of the content is equal to or greater than a predetermined threshold value, a bounding box may be set. For example, if the average value of pixel values corresponding to R of all pixels of the content is 630, the average value of pixel values corresponding to G is 530, and the average value of pixel values corresponding to B is 30, the preset The threshold may be 30.

The average of the pixel values may be calculated as in Equation 1 below.

[Equation 1]

In Equation 1, k is the number of pixels of the content, VRn is a pixel value corresponding to R of the nth pixel of the content, VGn is a pixel value corresponding to G of the nth pixel of the content, and VBn is n of the content It may be a pixel value corresponding to B of the th pixel. n is an integer greater than or equal to 1 and may be an integer from 1 to k.

The input unit 650 may receive information for the user to select a specific object when the user wants to obtain information on a specific object within the content. For example, when a first object and a second object exist in the content, the user may select the first object to obtain information on the first object. At this time, the user selects the first object through the input unit 650. You can choose. In this case, the user may select an object within the content by directly touching it using an external device (eg, mouse, keyboard, etc.) or using a touch function.

7 is a diagram illustrating a method of providing information on a specific object within content according to an embodiment of the present invention.

FIG. 7(a) is a diagram showing that content is displayed on a user device, and FIG. 7(b) is a diagram showing that information about a specific object selected by a user in content is displayed on a user device.

The content displayed on the user device shown in FIGS. 7 (a) and (b) includes content provided from the outside (eg, real-time video, image, audio, etc. provided through a broadcasting platform) or video or image captured by the user. Alternatively, it may be audio (speech) recorded by the user in real time through the user device.

The user device may extract all or some objects in the content. Referring to FIG. 7(a) , the user device may extract a first object 710, a second object 720, a third object 730, and a fourth object 740. For example, the first object 710 may be a chair, the second object 720 may be a laptop computer, the third object 730 may be a person, and the fourth object 740 may be a desk.

Referring to FIG. 7( b ), a user may select a specific object to obtain information from among objects extracted from content. For example, the user may select the second object 720 (eg, a notebook).

The user device may transmit a specific object (eg, a second object) selected by the user to the object information providing server. In other words, the user device may transmit information for identifying a specific object to the object information providing server. For example, the user device may separately crop a specific object in the content and transmit only the cropped portion to the server.

The object information providing server may transmit information about a specific object obtained by using the information for identifying the specific object received from the user device to the user device. In this case, information on a specific object may be directly obtained by an object information providing server, may be provided from an external search server (eg, a search portal site), or may be directly provided from a content provider.

Upon receiving information on a specific object from the object information providing server, the user device may display information on the specific object. For example, when the specific object selected by the user is a second object (eg, a laptop computer), the user device may display information 750 on the second object. In this case, the information 750 on the second object may be displayed overlapping with the content or displayed through a separate pop-up window. For example, when a user places a mouse cursor on a specific object, information on the specific object may be displayed around the mouse cursor. Information on a specific object may be configured differently according to the specific object selected by the user. For example, when a specific object is a thing, information on the specific object may include a model name, a manufacturing date, a color, a price (minimum or maximum price), a place of purchase (a link to a place of purchase), and the like. If a specific object is a person, information on the specific object may include a person's name, date of birth, a place where items worn by the person (eg, clothes, watches, shoes, etc.) can be purchased (link to a place of purchase), filmography, and the like. In addition, information on a specific object may be provided to the user in all forms that can be digitized, such as image, video, voice, as well as text format.

Whether information on a specific object is displayed overlaid on the content or displayed through a separate pop-up window is determined based on the data size of the information on the specific object received from the object information providing server or It can be determined based on what form the information is provided to the user. Specifically, when information on a specific object is provided as an image or video, the information on the specific object may be displayed in a pop-up form. Also, when the data size of information on a specific object is larger than a preset threshold value, it may be displayed in a pop-up form. Also, a location where information about a specific object is displayed may be determined according to a location where a specific object exists in content. In detail, when information about a specific object is overlapped and displayed on content, a position where information about a specific object is displayed may be determined based on the height or width of the content (or the height or width of a frame in the case of an image). For example, when the height of the content is H and a specific object exists below H/2 in the content, information about the specific object may be displayed above the specific object, and the specific object may be displayed above H/2 in the content. When present above 2, information on a specific object may be displayed below the specific object. Similarly, when the height of content is W, and a specific object exists on the left side of W/2 within the content, information on the specific object can be displayed on the right side of the specific object, and the specific object is located on the right side of W/2 within the content. When present in , information on a specific object may be displayed on the left side of the specific object.

Also, a position where information on a specific object is displayed may be determined by considering both the height and width of the content. For example, when a specific object exists below H/2 and to the left of W/2 in the content, information on the specific object may be displayed on the right side of the specific object. When a specific object exists below H/2 and to the right of W/2 in the content, information on the specific object may be displayed on the upper left side of the specific object. When a specific object exists above H/2 and to the left of W/2 in the content, information on the specific object may be displayed on the right and bottom of the specific object. When a specific object exists above H/2 and to the right of W/2 in the content, information on the specific object may be displayed on the left and bottom of the specific object.

Meanwhile, the user device may play audio content, and at this time, the user device may display the sound of the audio content as a subtitle. When a user selects a word for which information is to be obtained from subtitles, the user may be provided with information on the corresponding word. In this case, information on the corresponding word may be provided in the same way as information on the specific object described above.

Meanwhile, the user device may provide information on the recorded voice content to the user by taking the voice content recorded by the user in real time as an input. For example, the recorded voice content may be a word indicating a specific place or a specific object from which the user wants to obtain information. In this case, information on the recorded voice content may be provided in the same way as information on the specific object described above.

8 is a diagram illustrating a method of providing information on a specific object within content according to an embodiment of the present invention.

8(a) is a diagram illustrating that a user directly photographs a specific object for obtaining information with a user device. 8(b) is a diagram illustrating an image directly captured by a user. 8(c) is a diagram showing information on a specific object in an image directly photographed by a user.

Referring to FIG. 8(a) , there may be cases in which a user wants to obtain information about a specific object in daily life. Referring to FIG. 8(b) , a user may photograph content (eg, an image) including a plurality of objects including a specific object for which information is to be obtained. For example, a user may capture an image of a plurality of flowers including a specific flower in order to obtain information on a specific flower. At this time, since a plurality of flowers exist in the photographed image, the user device must extract each of the plurality of flowers in the photographed image. As shown in FIG. 8( b ), the user device may extract a plurality of flowers as a first flower 810 and a second flower 820, respectively, and may provide a separate bounding box. A user may select an object to obtain information from among a plurality of objects. In this case, a method for the user to select an object may be to use an external device (eg, a keyboard or a mouse) or a touch function as described above. After transmitting the specific object selected by the user to the object information providing server, the user device may acquire information about the specific object from the object information providing server. For example, referring to FIG. 8(c) , when the user selects the first flower 810, the user device may obtain information on the first flower 810 from the object information providing server and provide the information to the user. there is. At this time, the provided information may vary according to the type of a specific object selected by the user. For example, when a specific object is a flower, the provided information may include a plant name, flowering time, place of origin, purchase price, and place of purchase. Also, as described above, the provided information may be displayed overlapped within the content or displayed through a separate pop-up window.

9 is a flowchart illustrating a method of providing information on a specific object within content according to an embodiment of the present invention.

On the other hand, based on the method of providing information on a specific object in the content described below based on FIG. 9, ⓐ when the mouse is placed on a person in a YouTube video, related information is automatically provided, or ⓑ in the case of audio information, content If you place the mouse on the subtitle, related information with high relevance is provided, ⓒ provided information is provided in all possible forms of digitization, from basic text to video, or ⓓ If you are curious about a place or object while walking on the street , it can be provided upon receiving a request for information through a smartphone button or voice question.

Hereinafter, a method of providing information on a specific object within content according to an embodiment of the present invention described through FIGS. 5 to 8 with reference to FIG. 9 will be described in detail.

The user device 903 can display content to the user. At this time, the content may be provided from the content providing server 904 (S910). The content may also be content directly photographed or recorded by the user. The user device 903 may extract objects existing in the content (S920). A user may select a specific object to obtain information from among objects extracted by the user device 903 . In this case, the user device 903 may assign and display a separate bounding box to each of the objects extracted for the user's object selection. In this case, as a method of extracting a specific object, a method of using respective pixel values of pixels of the content described above may be applied. The user device 903 may transmit specific object identification information about a specific object selected by the user to the object information providing server 902 (S930). The specific object identification information is for the object information providing server 902 to identify a specific object. The object information providing server 902 identifies a specific object through specific object identification information (S940). The specific object identification information may be obtained by cropping a specific object selected by the user from content. The object information providing server 902 may transmit identification information on a specific object identified using the specific object identification information to the search server 901 . The identification information may be a model name in the case of an object, a person name in the case of a person, a plant name in the case of a plant, and the like. The search server 901 transmits information on a specific object using the identification information to the object information providing server 902, and the object information providing server 902 having received the information transmits the specific object information to the user device ( 903) can be transmitted (S960, S970). The user device may display and provide specific object information to the user (S980). In this case, the specific object information may be displayed overlapping with the content as described above or displayed through a separate pop-up window.

Meanwhile, the content providing server 904 providing content may provide information on objects included in the content to the object information providing server 902 . In this case, the object information providing server may transmit information about an object corresponding to specific object identification information transmitted by the user device among information about objects provided by the content providing server 904 to the user device. That is, when the content providing server provides information on objects included in the content to the object information providing server 902, steps S950 and S960 may be omitted.

Specific object information, which is information about a specific object, may be a plurality of pieces of information rather than one piece of information. The object information providing server 902 may provide some or all of the plurality of pieces of information to the user device 903 according to the level of the user device 903 . When only some of the plurality of pieces of information are provided to the user device 903, other pieces of the plurality of pieces of information may be blur-processed and provided to the user device 903. The level of the user device 903 may be set according to the amount (ie, billing) provided by the user to the object information providing server 902 .

The specific object information may include an internet address of a shopping mall where a specific object can be purchased. At this time, there may be a plurality of shopping malls, and the object information providing server 902 may receive a shopping mall connection fee from a specific shopping mall among the plurality of shopping malls, include only the Internet address of the specific shopping mall in specific object information, or The Internet address of a specific shopping mall may be provided with the highest priority among the Internet addresses of the plurality of shopping malls.

The object information providing server 902 and user device 903 shown in FIG. 9 may be the same as the object information providing server 520 and user device 530 of FIG. 5 .

In addition, the object detection device 400 according to an embodiment of the present invention may include the following features.

The object detection device 400 may include, for example, an input unit for receiving image data including an object; an extraction unit 410 for extracting at least one feature point of the object from the input image data; a feature generator 420 generating feature point description data corresponding to the extracted feature points; and an object detection unit 430 that identifies content corresponding to the image data based on the generated feature point description data.

The feature generator 420 includes: a region determiner for determining a plurality of regions around the extracted feature points; a size determiner configured to determine a size of data describing each of the determined areas based on a distance between each of the determined areas and the extracted feature point; and a generating unit generating the feature point description data based on the size of the determined data. can include

The size of the data may correspond to, for example, an integer value or a decimal point number of a feature vector between the feature point and each of the determined regions.

The object detection device 400 may receive image data including a specific object for requesting specific content from the user device 530 .

The object detection device 400 may extract feature points of an object included in the image data received from the user device 530 and identify content corresponding to the image data using the extracted feature points. For example, the identified content may be plural.

At this time, the object detection device 400 transmits the identified content information to the user device 530, so that the user device 530 requests the object information providing server 520 for the corresponding content. Alternatively, the object detection device 400 may transmit the identified content information to the object information providing server 520 so that the object information providing server 520 transmits the corresponding content to the user device 530 .

The object information providing server 520 may receive a request for specific content from the user device 530 and transmit the corresponding content to the user device 530 in response to the content request. In addition, the object information providing server 520 may receive a request for transmission of specific content from the object detection device 400 to the user device 530, and at this time, the object information providing server 520 responds to the content transmission request. With this, the corresponding content can be transmitted to the user device 530 .

When the object information providing server 520 receives a request for transmission of specific content from the object detection device 400 to the user device 530, the object information providing server 520 does not immediately transmit the content to the user device 530, but from the object detection device 400 By transmitting data corresponding to the received content information to the user device 530, it is possible to receive confirmation whether the content desired by the user is correct. When the object detecting device 400 identifies a plurality of contents, the user may select the contents to be viewed from among the plurality of contents.

The content identification service may be provided through an application installed on the user device 530 . Here, the application means an application, and may include, for example, an app running on a smartphone.

The input unit may receive image data including an object from the user device 530 .

The extraction unit 410 may extract at least one feature point of an object included in image data input from the input unit.

In addition, the extractor 410 may extract at least one feature point representing a unique feature of image data input from the input unit using a feature point extraction algorithm. In this case, the feature point extraction algorithm may be either a Speed Up Robust Feature Transform (SURF) algorithm or a Scale Invariant Feature Transform (SIFT) algorithm.

The feature generator 420 may generate feature point description data corresponding to the feature points extracted by the extractor 410 .

Also, the feature generator 420 may include a region determiner, a size determiner, and a generator.

The region determiner may determine a plurality of regions around the feature points extracted by the extractor 410 . For example, the plurality of regions around the feature points may be 16 4x4 regions, which is just an example, and there may actually be a plurality of regions of various sizes or numbers as well as 4x4 regions.

The size determination unit may determine the size of data describing each region determined by the region determination unit based on the distance between each region determined by the region determination unit and the feature point extracted by the extraction unit 410 . At this time, when the first area, which is one of a plurality of areas around the feature point, is closer to the feature point than the second area, the size determining unit determines the size of data describing the first area to be greater than the size of data describing the second area. can

In addition, the size of the data may correspond to an integer value or decimal point number of a feature vector between each feature point and the determined region. That is, the length of the descriptor of the feature point may be kept short by limiting the integer value or the number of decimal places in the distance between the feature point and the specific region.

In the prior art, all 16 areas have the same data size (weight), but the size determination unit allocates 4 bytes to each area by assigning weights to 4 areas in the middle near the feature point, and allocates 2 bytes to 12 areas on the remaining border. can In addition, the size determiner may allocate 2 bytes to 4 vertex areas far from the feature point and allocate 4 bytes to each of the remaining 12 areas relatively short from the feature point. Similarly, the size determiner allocates 4 bytes to each area by assigning weights to the 4 regions in the middle close to the feature point, allocates 2 bytes to the 4 vertex regions far from the feature point, and allocates 3 bytes to the remaining 8 regions at an appropriate distance from the feature point. can be assigned.

The generation unit may generate feature point description data based on the size of the data determined by the size determination unit.

The generation unit may generate feature point description data using the orientation of the extracted feature points and edge histograms for a plurality of regions determined by the region determination unit.

The object detector 430 may identify content corresponding to the image data received from the input unit based on the feature point description data generated by the feature generator 420 . At this time, the object detection unit 430 may identify a plurality of contents.

The communication unit may transmit information about the content identified by the object detection unit 430 to the user device 530 or the object information providing server 520 . At this time, the object information providing server 520 may transmit data corresponding to the transmitted content information to the user device 530 . Here, the data corresponding to the content information may be data used by the user device 530 to determine whether the content of the image received from the input unit matches the content identified by the object detection unit 430 .

Data corresponding to information of content may be identified content or advertisement information for the identified content. Also, the data corresponding to the content information may be result information of an arbitrary search engine for the identified content.

In addition, data corresponding to content information may include a signal for executing a specific application installed in the user device 530 .

When the object detector 430 identifies a plurality of contents, the communication unit may transmit information on the identified plurality of contents to the object information providing server 520 . At this time, the object information providing server 520 may transmit a plurality of data corresponding to the transmitted plurality of content information to the user device 530 . Also, the communication unit may receive a signal for selecting one of a plurality of pieces of data corresponding to information on a plurality of contents from the user device 530 .

In this case, content information on the selected data among the plurality of data may be matched with image data input from the input unit and stored in the database.

Databases store data. At this time, the data includes data input and output between components inside the object detection device 400, and data input and output between the object detection device 400 and components outside the object detection device 400. contains data For example, the database may store location information of at least one extraction point extracted by the extraction unit 410 . An example of such a database includes a hard disk drive, a read only memory (ROM), a random access memory (RAM), a flash memory, and a memory card existing inside or outside the object detecting device 400 .

An electronic device (ie, an object detection device, an object information providing server, and/or a user device) according to various embodiments of the present document may include, for example, a smart phone, a tablet PC, a mobile phone, a video phone, and an e-book reader. , a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a PDA, a portable multimedia player (PMP), an MP3 player, a medical device, a camera, or a wearable device. A wearable device may be in the form of an accessory (e.g. watch, ring, bracelet, anklet, necklace, eyeglasses, contact lens, or head-mounted-device (HMD)), integrated into textiles or clothing (e.g. electronic garment); In some embodiments, the electronic device may include, for example, a television, a digital video disk (DVD) player, Audio, refrigerator, air conditioner, vacuum cleaner, oven, microwave, washing machine, air purifier, set top box, home automation control panel, security control panel, media box (e.g. Samsung HomeSync ^TM , Apple TV ^TM , or Google TV ^TM ) , a game console (eg, Xbox ^TM , PlayStation ^TM ), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

In another embodiment, the electronic device may include various types of medical devices (e.g., various portable medical measuring devices (such as blood glucose meter, heart rate monitor, blood pressure monitor, or body temperature monitor), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), CT (computed tomography), imager, or ultrasonicator, etc.), navigation device, global navigation satellite system (GNSS), EDR (event data recorder), FDR (flight data recorder), automobile infotainment device, marine electronic equipment (e.g. navigation devices for ships, gyrocompasses, etc.), avionics, security devices, head units for vehicles, industrial or home robots, drones, ATMs in financial institutions, point of sale (POS) in stores of sales), or IoT devices (eg, light bulbs, various sensors, sprinkler devices, fire alarms, thermostats, street lights, toasters, exercise equipment, hot water tanks, heaters, boilers, etc.). According to some embodiments, the electronic device may be a piece of furniture, a building/structure or a vehicle, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (eg, water, electricity, gas, radio wave measuring device, etc.). In various embodiments, the electronic device may be flexible or a combination of two or more of the various devices described above. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices. In this document, the term user may refer to a person using an electronic device or a device using an electronic device (eg, an artificial intelligence electronic device).

Also, the electronic device may include a bus, a processor, a memory, an input/output interface, a display, and a communication interface. In some embodiments, the electronic device may omit at least one of the components or may additionally include other components. A bus may include circuitry that connects components together and carries communications (eg, control messages or data) between components. The processor may include one or more of a central processing unit, an application processor, or a communication processor (Communication Processor (CP)). The processor may, for example, execute calculations or data processing related to control and/or communication of at least one other element of the electronic device.

Memory may include volatile and/or non-volatile memory. The memory may store, for example, commands or data related to at least one other component of the electronic device. According to one embodiment, the memory may store software and/or programs. Programs may include, for example, kernels, middleware, application programming interfaces (APIs), and/or application programs (or "applications"). At least part of a kernel, middleware, or API may be referred to as an operating system. The kernel, for example, controls or controls system resources (eg, a bus, processor, or memory, etc.) used to execute operations or functions implemented in other programs (eg, middleware, APIs, or application programs). can manage Also, the kernel may provide an interface capable of controlling or managing system resources by accessing individual components of the electronic device in middleware, API, or application programs.

Middleware, for example, may perform an intermediary role so that an API or an application program communicates with a kernel to exchange data. Also, the middleware may process one or more task requests received from the application program according to priority. For example, the middleware may assign a priority for using system resources (eg, a bus, processor, memory, etc.) of the electronic device to at least one of the application programs and process the one or more task requests. An API is an interface for an application to control functions provided by the kernel or middleware, and includes at least one interface or function (eg, command) for file control, window control, image processing, or text control, for example. can do. The input/output interface transfers, for example, commands or data input from a user or other external device to other component(s) of the electronic device, or commands or data received from other component(s) of the electronic device. It can be output to users or other external devices.

The display may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a microelectromechanical system (MEMS) display, or an electronic paper display. there is. The display may display, for example, various contents (eg, text, image, video, icon, and/or symbol) to the user. The display may include a touch screen, and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a part of the user's body. The communication interface may establish communication between the electronic device and an external device (eg, a first external electronic device, a second external electronic device, or a server). For example, the communication interface may be connected to a network through wireless communication or wired communication to communicate with an external device (eg, a second external electronic device or server). Here, the hovering input identifies the approach of the electronic pen or a part of the user's body through the display by static electricity, even though the electronic pen or part of the user's body is not physically (and/or directly) in contact with the display ( and/or recognition), and may include, for example, hovering coordinate information.

Wireless communication is, for example, LTE, LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), or GSM (Global System for Mobile Communications) may include cellular communication using at least one of the like. According to one embodiment, wireless communication, for example, WiFi (wireless fidelity), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, near field communication (NFC), magnetic secure transmission (Magnetic Secure Transmission), radio It may include at least one of a frequency (RF) and a body area network (BAN). According to one embodiment, wireless communication may include GNSS. The GNSS may be, for example, a Global Positioning System (GPS), a Global Navigation Satellite System (Glonass), a Beidou Navigation Satellite System (hereinafter “Beidou”) or Galileo, the European global satellite-based navigation system. Hereinafter, in this document, "GPS" may be used interchangeably with "GNSS". Wired communication may include, for example, at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), power line communication, or plain old telephone service (POTS). there is. The network may include at least one of a telecommunications network, for example, a computer network (eg, LAN or WAN), the Internet, or a telephone network.

Each of the first and second external electronic devices may be the same or different type of the electronic device. According to various embodiments, all or part of operations executed in an electronic device may be executed in one or more electronic devices (eg, an electronic device or a server). According to an embodiment, an electronic device may perform certain functions or services. When it is necessary to automatically or upon request, the electronic device may request at least some functions related to it from another device (eg, an electronic device or a server) instead of or in addition to executing the function or service by itself. Another electronic device (eg, an electronic device or a server) may execute the requested function or additional function and deliver the result to the electronic device, which may process the received result as it is or additionally, and perform the requested function For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

Electronic devices include one or more processors (eg, APs), communication modules, (subscriber identification modules, memory, sensor modules, input devices, displays, interfaces, audio modules, camera modules, power management modules, batteries, indicators, and motors). The processor may, for example, drive an operating system or an application program to control a plurality of hardware or software components connected to the processor, and may perform various data processing and calculations. For example, it may be implemented as a system on chip (SoC). According to one embodiment, the processor may further include a graphic processing unit (GPU) and/or an image signal processor. The processor may include other components (eg : A command or data received from at least one of the non-volatile memory) may be loaded into the volatile memory, processed, and resultant data may be stored in the non-volatile memory.

It may have the same or similar configuration as the communication module (eg, communication interface). The communication module may include, for example, a cellular module, a WiFi module, a Bluetooth module, a GNSS module, an NFC module, and an RF module. The cellular module may provide, for example, a voice call, a video call, a text service, or an Internet service through a communication network. According to one embodiment, the cellular module may perform identification and authentication of an electronic device within a communication network using a subscriber identification module (eg, a SIM card). According to one embodiment, the cellular module may perform at least some of the functions that a processor can provide. According to one embodiment, the cellular module may include a communication processor (CP). According to some embodiments, at least some (eg, two or more) of the cellular module, WiFi module, Bluetooth module, GNSS module, or NFC module may be included in one integrated chip (IC) or IC package. The RF module may transmit and receive communication signals (eg, RF signals), for example. The RF module may include, for example, a transceiver, a power amp module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of a cellular module, a WiFi module, a Bluetooth module, a GNSS module, or an NFC module may transmit and receive an RF signal through a separate RF module. The subscriber identification module may include, for example, a card or an embedded SIM including the subscriber identification module, and may include unique identification information (eg, integrated circuit card identifier (ICCID)) or subscriber information (eg, international mobile terminal (IMSI)). subscriber identity)).

The memory (eg, memory) may include, for example, a built-in memory or an external memory. Built-in memory includes, for example, volatile memory (e.g. DRAM, SRAM, or SDRAM, etc.), non-volatile memory (e.g. one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, flash memory) , hard drive, or solid state drive (SSD) external memory may include a flash drive (flash drive), for example, CF (compact flash), SD (secure digital), Micro-SD, It may include a Mini-SD, extreme digital (xD), multi-media card (MMC), memory stick, etc. The external memory may be functionally or physically connected to an electronic device through various interfaces.

The sensor module may, for example, measure a physical quantity or detect an operating state of an electronic device, and convert the measured or sensed information into an electrical signal. The sensor module may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor (eg, a red, green, blue (RGB) sensor), and a biosensor. , a temperature/humidity sensor, an illuminance sensor, or an ultra violet (UV) sensor. Additionally or alternatively, the sensor module may include, for example, an e-nose sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, ) sensor, an iris sensor and/or a fingerprint sensor. The sensor module may further include a control circuit for controlling one or more sensors included therein. In some embodiments, the electronic device may further include a processor configured to control the sensor module, either as part of the processor or separately, to control the sensor module while the processor is in a sleep state.

The input device may include, for example, a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device. The touch panel may use at least one of, for example, a capacitive type, a pressure-sensitive type, an infrared type, or an ultrasonic type. Also, the touch panel may further include a control circuit. The touch panel may further include a tactile layer to provide a tactile response to the user. The (digital) pen sensor may be, for example, a part of the touch panel or may include a separate recognition sheet. Keys may include, for example, hardware buttons, optical keys, or keypads. The ultrasonic input device may detect ultrasonic waves generated by an input tool through a microphone and check data corresponding to the detected ultrasonic waves.

The display may include a panel, a hologram device, a projector, and/or control circuitry for controlling them. The panel may be implemented to be flexible, transparent, or wearable, for example. The panel may be composed of a touch panel and one or more modules. According to one embodiment, the panel may include a pressure sensor (or force sensor) capable of measuring the strength of a user's touch. The pressure sensor may be implemented integrally with the touch panel, or may be implemented as one or more sensors separate from the touch panel. A hologram device can show a 3D image in the air by using the interference of light. The projector may display an image by projecting light onto a screen. The screen may be located inside or outside the electronic device, for example. The interface may include, for example, HDMI, USB, optical interface, or D-subminiature (D-subminiature) 278 . Additionally or alternatively, the interface may include, for example, a mobile high-definition link (MHL) interface, an SD card/multi-media card (MMC) interface, or an infrared data association (IrDA) standard interface.

The audio module can, for example, convert a sound and an electrical signal in both directions. The audio module may process sound information input or output through, for example, a speaker, receiver, earphone, or microphone. The camera module is, for example, a device capable of capturing images (eg, static images and/or dynamic images) and moving images, and according to one embodiment, one or more image sensors (eg, a front sensor or a rear sensor), a lens , an image signal processor (ISP), or a flash (eg LED or xenon lamp, etc.). The power management module may manage power of the electronic device, for example. According to one embodiment, the power management module may include a power management integrated circuit (PMIC), a charging IC, or a battery or fuel gauge. A PMIC may have a wired and/or wireless charging method. The wireless charging method includes, for example, a magnetic resonance method, a magnetic induction method, or an electromagnetic wave method, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonance circuit, or a rectifier. there is. The battery gauge may measure, for example, remaining capacity of the battery, voltage, current, or temperature during charging. Batteries may include, for example, rechargeable cells and/or solar cells.

The indicator may indicate a specific state of the electronic device or a part thereof (eg, a processor), for example, a booting state, a message state, or a charging state. The motor may convert electrical signals into mechanical vibrations and generate vibrations or haptic effects. The electronic device is, for example, a mobile TV support device (eg, GPU) capable of processing media data according to standards such as digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or mediaFlo ^TM . can include Each of the components described in this document may be composed of one or more components, and the name of the corresponding component may vary depending on the type of electronic device. In various embodiments, an electronic device (eg, an electronic device) is configured as a single entity by omitting some components, further including additional components, or combining some of the components, but The functions of the components can be performed identically.

In various embodiments of the present disclosure, an electronic device (or electronic device) may include a housing including a front side, a rear side, and a side surface surrounding a space between the front side and the back side. A touch screen display (eg, display) is disposed within the housing and may be exposed through the front surface. A microphone is disposed within the housing and may be exposed through a portion of the housing. At least one speaker is disposed within the housing and may be exposed through another part of the housing. A hardware button (eg key) may be placed on another part of the housing or set to display on the touchscreen display. A wireless communication circuit (eg, a communication module) may be located within the housing. The processor (or processor) may be located in the housing and electrically connected to the touch screen display, the microphone, the speaker, and the wireless communication circuit. The memory (or memory) may be located in the housing and electrically connected to the processor.

In various embodiments of the present invention, the memory is set to store a first application program including a first user interface for receiving text input, and the memory, when executed, causes the processor to perform the first operation and instructions that cause a second operation to be performed, wherein the first operation receives a first type of user input through the button while the first user interface is not displayed on the touch screen display; , after receiving the first type of user input, receiving a first user utterance through the microphone, and to an external server including an automatic speech recognition (ASR) and intelligence system; At least one method for providing first data for a first user utterance and, after providing the first data, performing a task generated by the intelligent system in response to the first user utterance from the external server. A command is received, and the second operation is performed after receiving the first user input through the button while the first user interface is displayed on the touch screen display and receiving the first type of user input. In this case, after receiving the second user speech through the microphone, providing second data for the second user speech to the external server, and providing the second data, from the server, the second user Data on text generated by the automatic speech recognition from utterances may be received, but the text may be input to the first user interface without receiving a command generated by the intelligent system.

In various embodiments of the present invention, the button may include a physical key located on the side of the housing.

In various embodiments of the present invention, the first type of user input is after pressing the button once, pressing the button twice, pressing the button three times, or pressing the button once. It can be either a held press, or a two-time press and hold press on the button.

In various embodiments of the present disclosure, the instructions may further cause the processor to display the first user interface along with a virtual keyboard. The button may not be part of the virtual keyboard.

In various embodiments of the invention, the instructions may further cause the processor to receive, from the external server, data for text generated by ASR from the first user utterance within the first action. there is.

In various embodiments of the present disclosure, the first application program may include at least one of a note application program, an e-mail application program, a web browser application program, or a calendar application program.

In various embodiments of the present invention, the first application program includes a message application, and the instructions are performed by the processor when a selected time (and/or period) exceeds after inputting the text, the wireless It may further cause the automatically inputted text to be transmitted via the communication circuit.

In various embodiments of the present invention, the instructions further cause the processor to perform a third operation, wherein the third operation is performed by pressing the button while displaying the first user interface on the touchscreen display. After receiving the second type of user input, a third user speech is received through the microphone, and a third user speech response to the third user speech is received by the external server. After providing data and providing the third data, at least one command for performing a task generated by the intelligent system in response to the third user's utterance may be received from the external server.

In various embodiments of the present disclosure, the instructions further cause the processor to perform a fourth operation, wherein the fourth operation is performed while the first user interface is not displayed on the touch screen display. The second type of user input is received through a button, and after receiving the second type of user input, a fourth user speech is received through the microphone, and fourth data for the fourth user speech is received. Provided to the external server, and after providing the fourth data, in response to the fourth user utterance, receiving at least one command for performing a task generated by the intelligent system from the external server; Receiving a fifth user speech through the microphone, providing fifth data for the fifth user speech to the external server, and after providing the fifth data, in response to the fifth user speech At least one command for performing a task generated by the intelligent system may be received from the external server.

In various embodiments of the present invention, the first type of user input and the second type of user input are different, and include pressing the button once, pressing the button twice, and pressing the button three times. , Pressing and maintaining the button after pressing the button once, or pressing and maintaining the button twice and maintaining the button may be selected.

In various embodiments of the present invention, the memory is further configured to store a second application program including a second user interface for receiving text input, and the instructions, when executed, cause the processor to: further cause an operation to be performed, wherein the third operation is to receive the first type of user input through the button while displaying the second user interface, and after the first type of user input is received; , Receiving a third user speech through the microphone, providing third data for the third user speech to the external server, and after providing the third data, from the external server, the third data While receiving data on text generated by ASR from user utterances, but not receiving commands generated by the intelligent system, the text is input to the second user interface, and the selected time (and/or period) is If exceeded, the input text may be automatically transmitted through the wireless communication circuit.

In various embodiments of the present invention, the memory is set to store a first application program including a first user interface for receiving text input, and the memory, when executed, causes the processor to perform the first operation and instructions that cause the second operation to be performed, the first operation comprising: receiving a first type of user input through the button, receiving a first user utterance through the microphone after receiving the first type of user input, automatic speech recognition (ASR), and An external server including an intelligence system, providing first data for the first user utterance, and after providing the first data, in response to the first user utterance, by the intelligence system At least one command for performing the created task is received from the external server, and the second operation is performed after receiving a second type of user input through the button and receiving the second type of user input. In this case, after receiving the second user speech through the microphone, providing second data for the second user speech to the external server, and providing the second data, from the server, the second user While receiving data on text generated by ASR from the utterance, the command generated by the intelligent system is not received, and the text can be input to the first user interface.

In various embodiments of the present disclosure, the instructions may further cause the processor to display the first user interface along with a virtual keyboard, and the buttons may not be part of the virtual keyboard.

In various embodiments of the invention, the instructions may further cause the processor to receive data for text generated by the ASR from the first user utterance within the first operation from the external server. .

In various embodiments of the present disclosure, the first application program may include at least one of a note application program, an e-mail application program, a web browser application program, or a calendar application program.

In various embodiments of the present invention, the first application program includes a message application, and the instructions are performed by the processor when a selected time (and/or period) exceeds after inputting the text, the wireless It may further cause the automatically inputted text to be transmitted via the communication circuit.

In various embodiments of the present invention, the instructions may further cause the processor to perform the first operation independently of display on the display of the first user interface.

In various embodiments of the present disclosure, the instructions further cause the processor to perform the second operation when at least one of the electronic device is locked or the touchscreen display is turned off. can do.

In various embodiments of the present disclosure, the instructions may further cause the processor to perform the second operation while displaying the first user interface on the touch screen display.

In various embodiments of the present invention, the memory, when executed, allows the processor to receive user utterances through the microphone, perform automatic speech recognition (ASR) or natural language understanding (NLU) To an external server that performs at least one of the following: Transmits, along with data on the user's speech, information associated with whether or not to perform the natural language understanding on text obtained by performing the ASR on the data on the user's speech; and , if the information indicates not to perform the natural language understanding, receive the text for data on the user utterance from the external server, and if the information indicates that the natural language understanding is not to be performed, the text from the external server It is possible to store an instruction that causes to receive a command obtained as a result of performing the natural language understanding for .

According to one embodiment, a program module (eg, program) may include an operating system that controls resources related to an electronic device (eg, the electronic device) and/or various applications (eg, application programs) running on the operating system. there is. The operating system may include, for example, Android ^TM , iOS ^TM , Windows ^TM , Symbian ^TM , Tizen ^TM , or Bada ^TM . The program module may include a kernel (eg, kernel), middleware (eg, middleware), (API (eg, API), and/or an application (eg, application program). At least a portion of the program modules may be configured on an electronic device). It can be preloaded on or downloaded from an external electronic device (eg, electronic device, server, etc.).

The kernel may include, for example, system resource managers and/or device drivers. A system resource manager can control, allocate, or reclaim system resources. According to one embodiment, the system resource manager may include a process management unit, a memory management unit, or a file system management unit. The device driver may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WiFi driver, an audio driver, or an inter-process communication (IPC) driver. Middleware, for example, may provide functions commonly required by applications or provide various functions to applications through an API so that applications can use limited system resources inside the electronic device. According to one embodiment, the middleware is a runtime library, application manager, window manager, multimedia manager, resource manager, power manager, database manager, package manager, connectivity manager, notification manager, location manager, graphics manager, or security manager. may contain at least one.

A runtime library may include, for example, a library module used by a compiler to add new functions through a programming language while an application is running. A runtime library can perform I/O management, memory management, or processing of arithmetic functions. The application manager may manage the life cycle of the application, for example. A window manager can manage GUI resources used in a screen. The multimedia manager can identify a format required for reproducing media files, and encode or decode the media file using a codec suitable for the format. A resource manager can manage an application's source code or memory space. The power manager may manage, for example, battery capacity or power, and provide power information necessary for the operation of the electronic device. According to one embodiment, the power manager may interoperate with a basic input/output system (BIOS). A database manager can create, search, or change a database to be used in an application, for example. The package manager may manage installation or update of applications distributed in the form of package files.

A connectivity manager can manage wireless connections, for example. The notification manager may provide a user with an event such as an arrival message, an appointment, or proximity notification. The location manager may manage location information of the electronic device, for example. The graphic manager may manage, for example, a graphic effect to be provided to a user or a user interface related thereto. A security manager may provide system security or user authentication, for example. According to one embodiment, middleware may include a telephony manager for managing voice or video call functions of an electronic device or a middleware module capable of forming a combination of functions of the aforementioned components. According to one embodiment, middleware may provide modules specialized for each type of operating system. Middleware can dynamically delete some existing components or add new components. The API is, for example, a set of API programming functions, and may be provided in different configurations depending on the operating system. For example, in the case of Android or iOS, one API set can be provided for each platform, and in the case of Tizen, two or more API sets can be provided for each platform.

Applications include, for example, home, dialer, SMS/MMS, IM (instant message), browser, camera, alarm, contact, voice dial, email, calendar, media player, album, watch, health care (eg exercise or measurement of blood sugar, etc.), or environmental information (eg, atmospheric pressure, humidity, or temperature information) providing applications. According to one embodiment, the application may include an information exchange application capable of supporting information exchange between an electronic device and an external electronic device. The information exchange application may include, for example, a notification relay application for delivering specific information to an external electronic device or a device management application for managing an external electronic device. For example, a notification delivery application may transfer notification information generated by another application of an electronic device to an external electronic device or may receive notification information from an external electronic device and provide the notification information to a user. The device management application is, for example, a function of an external electronic device that communicates with the electronic device (eg, turn-on/turn-off of the external electronic device itself (or some components) or brightness (or resolution) of a display). adjustment), or an application operating in an external electronic device may be installed, deleted, or updated. According to one embodiment, the application may include an application designated according to the properties of the external electronic device (eg, a health management application of a mobile medical device). According to one embodiment, the application may include an application received from an external electronic device. At least some of the program modules may be implemented (eg, executed) in software, firmware, hardware (eg, a processor), or a combination of at least two of them, and may be implemented as modules, programs, routines, or instruction sets for performing one or more functions. or process.

The methods according to the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded on a computer readable medium. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer readable medium may be specially designed and configured for the present invention or may be known and usable to those skilled in computer software.

Examples of computer readable media may include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter and the like. The hardware device described above may be configured to operate with at least one software module to perform the operations of the present invention, and vice versa.

In addition, the above-described method or device may be implemented by combining all or some of its components or functions, or may be implemented separately.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

Claims (1)

A method and apparatus for analyzing images.

Images