KR102407815B1 - Apparatus and method for processing image - Google Patents

Abstract

Disclosed are an image processing apparatus and an image processing method for estimating and focusing a region of interest based on interest information learned by a data recognition model meeting a predetermined condition among a plurality of data recognition models.
In this case, the image processing apparatus may estimate the ROI using a rule-based or artificial intelligence algorithm. When the ROI is estimated using an artificial intelligence algorithm, the image processing apparatus may estimate the ROI using a machine learning, neural network, or deep learning algorithm.

Description

Image processing apparatus and method {Apparatus and method for processing image}

The present invention relates to an image processing apparatus and an image processing method.

In addition, the present disclosure relates to an artificial intelligence (AI) system for simulating functions such as cognition and judgment of the human brain by using a machine learning algorithm such as deep learning and an application technology thereof.

Automatically focusing an image in processing an image is a task that can increase the user's satisfaction with the image. There is an image processing method that focuses.

In addition, artificial intelligence systems have recently been introduced into the image processing field.

An artificial intelligence system is a computer system that implements human-level intelligence, and unlike the existing rule-based smart system, it is a system in which a machine learns, judges, and becomes smarter by itself. As artificial intelligence systems are used, the recognition rate improves and users can understand user preferences more accurately.

Artificial intelligence technology consists of machine learning (deep learning) and element technologies using machine learning.

Machine learning is an algorithm technology that categorizes/learns the characteristics of input data by itself, and element technology uses machine learning algorithms such as deep learning to simulate functions such as cognition and judgment of the human brain. It consists of technical fields such as understanding, reasoning/prediction, knowledge expression, and motion control.

The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing and applying/processing human language/text, and includes natural language processing, machine translation, dialogue system, question and answer, and speech recognition/synthesis. Visual understanding is a technology for recognizing and processing objects like human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image improvement, and the like. Inferential prediction is a technology for logically reasoning and predicting by judging information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data generation/classification) and knowledge management (data utilization). Motion control is a technology for controlling autonomous driving of a vehicle and movement of a robot, and includes motion control (navigation, collision, driving), manipulation control (action control), and the like.

An object of the present invention is to provide an image processing apparatus and an image processing method for estimating and focusing a region of interest based on interest information learned by a data recognition model meeting a predetermined condition among a plurality of data recognition models.

An image processing apparatus according to a first aspect includes: a photographing unit configured to acquire a live view image including at least one subject; a memory storing computer executable instructions; By executing the computer-executable instruction, based on a data recognition model corresponding to a predetermined condition among data recognition models, the obtained live view is based on a criterion for determining whether the data recognition model corresponds to the learned interest information. at least one processor for estimating a user's ROI from the image and focusing on the estimated ROI; and an input/output unit configured to display the live view image focused on the estimated ROI.

An image processing method according to a second aspect includes: acquiring a live view image including at least one subject; Based on a data recognition model corresponding to a predetermined condition among data recognition models, according to a criterion for determining whether the data recognition model corresponds to the learned interest information step; focusing on the estimated ROI; and displaying the live view image focused on the estimated ROI.

According to a third aspect, it is a computer-readable recording medium in which a program for executing the image processing method in a computer is recorded.

1A is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.
1B is a diagram illustrating a configuration of an image processing apparatus according to various embodiments of the present disclosure;
2 is a block diagram of a photographing unit according to an exemplary embodiment.
3 is a view for explaining an operation of a control unit according to an embodiment.
4A and 4B are diagrams for explaining an example in which a salient region is estimated as a user's ROI and focused in the image processing apparatus according to an exemplary embodiment.
5A and 5B are diagrams for explaining an example of estimating a region corresponding to personalized interest information as a user's region of interest in the image processing apparatus according to an embodiment of the present disclosure;
6 is a block diagram of a control unit according to an embodiment.
7 is a block diagram of a data learning unit according to an exemplary embodiment.
8 is a block diagram of a data recognition unit according to an exemplary embodiment.
9 is a block diagram illustrating an image processing apparatus according to another exemplary embodiment.
10 is a diagram illustrating an example of learning and recognizing data by interworking with an image processing apparatus and a server according to some embodiments.
11 is a block diagram illustrating an image processing apparatus according to another exemplary embodiment.
12 is a flowchart illustrating an image processing method according to an exemplary embodiment.
13 is a flowchart illustrating a situation of estimating a region of interest when an image processing apparatus includes a first processor and a second processor according to an exemplary embodiment.
14 is a flowchart illustrating a situation in which an ROI is estimated when an image processing apparatus includes a first processor, a second processor, and a third processor, according to an exemplary embodiment.
15 is a flowchart illustrating another situation in which an ROI is estimated when an electronic device includes a first processor, a second processor, and a third processor, according to an embodiment.
16 is a flowchart illustrating a situation in which an image processing apparatus estimates a region of interest using a server, according to an exemplary embodiment.

Hereinafter, an exemplary embodiment will be described in detail with reference to the accompanying drawings. Of course, the following examples are not intended to limit or limit the scope of rights only for the purpose of specifying the technical content. What can be easily inferred by an expert in the technical field from the detailed description and examples is construed as belonging to the scope of the right.

On the other hand, in the present specification, when a component is "connected" with another component, this includes not only a case of 'directly connected' but also a case of 'connected with another component interposed therebetween'. In addition, when a component "includes" another component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

Also, terms including ordinal numbers such as 'first' or 'second' used in this specification may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

As used herein, the term “image processing device” refers to a generic term for an electronic device having a photographing function. For example, a device such as a smartphone or a digital camera having a camera module may correspond to the image processing apparatus.

The present embodiments relate to an image processing apparatus and an image processing method, and detailed descriptions of matters widely known to those of ordinary skill in the art to which the following embodiments belong will be omitted.

1A is a block diagram illustrating an image processing apparatus 1000 according to an exemplary embodiment. Those of ordinary skill in the art related to the present embodiment can see that other general-purpose components other than those shown in FIG. 1 may be further included.

As shown in FIG. 1A , the image processing apparatus 1000 according to an embodiment may include a memory 1100 , a controller 1200 , an input/output unit 1300 , and a photographing unit 1610 .

The photographing unit 1610 may acquire an image including at least one subject. For example, the photographing unit 1610 may acquire a live view image including at least one subject, and may acquire a captured image to be stored in the image processing apparatus 1000 when actual photographing is performed. The photographing unit 1610 may photograph an image by focusing on the estimated ROI in response to the user's photographing command. A detailed description of the photographing unit 1610 will be described in detail with reference to FIG. 2 .

The memory 1100 may store a program for processing and control by the controller 1200 , and may also store data input to or output from the image processing apparatus 1000 . Memory 1100 may store computer executable instructions.

The controller 1200 generally controls the overall operation of the image processing apparatus 1000 . The controller 1200 may include at least one processor. The control unit 1200 may include a plurality of processors or a single processor in an integrated form according to its function and role.

At least one processor constituting the control unit 1200 executes computer-executable instructions stored in the memory 1100, so that the data recognition model learns based on a data recognition model corresponding to a predetermined condition among the data recognition models. The user's ROI may be estimated from the acquired live-view image according to a criterion for determining whether the information corresponds to the ROI, and the estimated ROI may be focused. A detailed description of the control unit 1200 will be described in detail with reference to FIGS. 3 to 8 .

The input/output unit 1300 may display a live view image focused on the estimated ROI. The input/output unit 1300 may display a live view image in which the focusing operation on the region of interest is reflected in real time.

1B is a diagram illustrating a configuration of an image processing apparatus 1000 according to various embodiments of the present disclosure.

Referring to FIG. 1B , the image processing apparatus 1000 may include a controller 1200 having a first processor 1200a and a second processor 1200b.

The first processor 1200a controls execution of at least one application installed in the image processing apparatus 1000 , and provides graphics for images (eg, live view images, captured images, etc.) acquired by the image processing apparatus 1000 . processing can be performed. The first processor 1200a may be implemented in the form of a system on chip (SoC) in which functions such as a central processing unit (CPU), a graphic processing unit (GPU), a communication chip, and a sensor are integrated. Also, the first processor 1200a may be referred to as an application processor (AP) in this specification.

The second processor 1200b may estimate the ROI of the image using the data recognition model.

Meanwhile, the second processor 1200b may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI) that performs a function of estimating a region of interest using a data recognition model. According to various embodiments, in the case of a data recognition model using visual understanding as an element technology, a dedicated hardware chip for artificial intelligence (AI) may include a GPU.

Also, the image processing apparatus 1000 may further include a third processor, a fourth processor, and the like that perform the same function as the second processor 1200b. In this case, each processor may perform an ROI estimation function using different data recognition models.

According to various embodiments of the present disclosure, a function performed by the first processor 1200a may be stored in the memory 1100 and performed for applications that perform various functions, and a function performed by the second processor 1200b may be performed for the OS of the image processing apparatus 1000 .

For example, the camera application may generate a live view image and determine a data recognition model corresponding to a predetermined condition. The camera application may transmit the determined data recognition model and information related to the ROI estimation request to the OS and/or the server located outside the image processing apparatus 1000 .

The OS and/or an external server may estimate the ROI by using the included data recognition model, respectively. In this case, the OS and/or an external server may focus on the estimated ROI. The present invention is not limited thereto.

2 is a block diagram of a photographing unit according to an exemplary embodiment. Those of ordinary skill in the art related to the present embodiment can see that other general-purpose components other than the components shown in FIG. 2 may be further included.

The photographing unit 1610 is a component that generates an image of an electrical signal from incident light, and includes a lens 1611 , a lens driving unit 1612 , an iris 1613 , an iris driving unit 1614 , an imaging device 1615 , and an imaging device. and a device control unit 1616.

The lens 1611 may include a plurality of groups and a plurality of lenses. The position of the lens 1611 is adjusted by the lens driver 1612 . The lens driver 1612 adjusts the position of the lens 1611 according to a control signal provided from the controller 1200 .

The opening/closing degree of the diaphragm 1613 is controlled by the diaphragm driver 1614 , and the amount of light incident to the imaging device 1615 is adjusted.

The optical signal passing through the lens 1611 and the diaphragm 1613 arrives at the light receiving surface of the imaging device 1615 to form an image of the subject. The imaging device 1615 may be a Charge Coupled Device (CCD) image sensor or a Complementary Metal Oxide Semiconductor Image Sensor (CIS) that converts an optical signal into an electrical signal. The sensitivity of the imaging device 1615 may be adjusted by the imaging device controller 1616 . The imaging device controller 1616 may control the imaging device 1615 according to a control signal automatically generated by an image signal input in real time or a control signal manually input by a user's manipulation.

The exposure time of the imaging device 1615 is controlled by a shutter (not shown). The shutter (not shown) includes a mechanical shutter for controlling the incidence of light by moving a shade, and an electronic shutter for controlling exposure by supplying an electrical signal to the imaging device 1615 .

The analog signal processing unit (not shown) may perform noise reduction processing, gain adjustment, waveform shaping, analog-to-digital conversion processing, and the like, on the analog signal supplied from the imaging device 1615 . The image signal output from the analog signal processing unit (not shown) may be input to the control unit 1200 . The image signal input to the controller 1200 may be a live view image through digital signal processing.

3 is a view for explaining an operation of a control unit according to an embodiment.

At least one processor constituting the control unit 1200 may acquire a live view image based on an image signal input from the photographing unit 1610 . At least one processor constituting the control unit 1200 acquires, based on a data recognition model corresponding to a predetermined condition among a plurality of data recognition models, according to a criterion for determining whether the data recognition model corresponds to the learned interest information The user's ROI may be estimated from the live view image, and the estimated ROI may be focused.

For example, when the first condition is satisfied, the at least one processor constituting the control unit 1200 selects the saliency region of the user according to a criterion for determining whether the first data recognition model corresponds to the learned saliency region. It can be estimated as a region of interest. The protrusion area refers to an area generally recognized as a place that is highlighted or has a characteristic in an image, and may be determined by a predetermined criterion regarding an area occupied by a subject in an image or a color distribution of the image. The first data recognition model may learn a criterion for determining whether it corresponds to the salient region. The first data recognition model may use a criterion for determining whether the salient region corresponds to the learned salient region when determining the salient region in the live view image acquired by the photographing unit 1200 .

4A and 4B are diagrams for explaining an example in which the image processing apparatus 1000 according to an exemplary embodiment is focused by estimating the salient region as the user's ROI.

Referring to FIG. 4A , the shape or area of a person passing by a flower field of a single color of the same variety is different from that of the flower field as a background, and the difference is noticeable because they wear clothes of a different color from the yellow flowers, so the image processing device A reference numeral 1000 may determine that a region corresponding to a person in the image corresponds to a salient region, and may focus by estimating a region corresponding to a person in the image as a region of interest.

Referring to FIG. 4B , in the image in which there is also a long fence between a long road and a forest, the image processing apparatus 1000 determines that an area corresponding to a red public phone box erected alone corresponds to a protruding area, It is possible to focus by estimating a region corresponding to a public telephone box in the ROI as a region of interest.

When the second condition is satisfied, the at least one processor constituting the control unit 1200 is an area corresponding to the personalized interest information according to a criterion for determining whether the second data recognition model corresponds to the learned personalized interest information. can be estimated as the user's ROI. The personalized interest information may be determined by predetermined statistics regarding user images stored in the image processing apparatus 1000 . The second data recognition model may learn about a criterion for determining whether it corresponds to the personalized interest information. When determining a region corresponding to the personalized interest information in the live view image acquired by the photographing unit 1200 , the second data recognition model may use a criterion for determining whether it corresponds to the learned personalized interest information.

5A and 5B are diagrams for explaining an example in which the image processing apparatus 1000 according to an exemplary embodiment is focused by estimating a region corresponding to the personalized ROI as the user's ROI.

Referring to FIG. 5A , in an image including a person and a car, when the personalized interest information is a car, the image processing apparatus 1000 may focus by estimating a region corresponding to the car as the region of interest in the image. . That is, in an image including a person and a car, it may be common to focus on a person, but when it is determined that the personalized interest information is a car, the area corresponding to the car is estimated as the user's area of interest to focus can match

Referring to FIG. 5B , in an image including multiple babies, when the personalized interest information is the user's own baby, the image processing apparatus 1000 estimates a region corresponding to the user's own baby as the region of interest in the image. so you can focus. That is, in an image containing multiple babies, it may be common to focus on all babies or the closest baby, but if the personalized interest information is determined to be your own baby, your own baby It is possible to focus by estimating the region corresponding to the region of interest of the user.

Meanwhile, the second condition is a case in which the number of images stored in the image processing apparatus 1000 is greater than the predetermined number, the reliability of the personalized interest information satisfies the predetermined condition, and the first condition is that the second condition is satisfied. It means that you are not satisfied. If the number of images stored in the image processing apparatus 1000 is not greater than the predetermined number or the reliability of the personalized interest information does not satisfy the predetermined condition, it is considered that the second condition is not satisfied, and the personalized information is not satisfied. Since the accuracy of the ROI is low, the ROI is estimated using the first data recognition model.

At least one processor constituting the control unit 1200 may estimate a region corresponding to the interest information having a high priority as the user's region of interest based on the priority of the interest information.

At least one processor constituting the controller 1200 may perform multi-focus by focusing on all of the plurality of ROIs when a plurality of ROIs are estimated. Also, when a plurality of regions of interest are estimated, the at least one processor constituting the controller 1200 may focus on the region of interest selected by the user from among the plurality of regions of interest.

At least one processor constituting the control unit 1200 may transmit a control signal for focusing on the estimated ROI to the photographing unit 1610 .

6 is a block diagram of the controller 1200 according to an embodiment.

Referring to FIG. 6 , the control unit 1200 according to some embodiments may include a data learning unit 1210 and a data recognition unit 1220 .

In order for the data recognition model (eg, the first data recognition model of FIG. 3 , the second data recognition model of FIG. 3 ) to estimate the user's area of interest in the image, the data learning unit 1210 may determine which area in the image corresponds to the interest information. You can learn to have a criterion for judging whether or not to do it. The data learning unit 1210 may learn that the data recognition model has a criterion for determining which data to use to determine whether it corresponds to the interest information and how to use the data to determine whether it corresponds to the interest information. The data learner 1210 acquires data to be used for learning and applies the acquired data to a data recognition model to learn a criterion for determining which region in the image corresponds to the information of interest.

The data recognizer 1220 may determine which region in the image corresponds to the interest information based on various types of data. The data recognition unit 1220 may determine which region in the image corresponds to the information of interest by using a data recognition model learned in response to a predetermined condition from among the data recognition models. The data recognition unit 1220 applies the data recognition model using a live view image including at least one subject as an input value according to a criterion obtained by learning for each data recognition model, so that a certain region in the image is identified as the information of interest. It can be determined whether Meanwhile, the result of estimating the user's region of interest by applying the data recognition model to determine which region in the image corresponds to the interest information and the user's response to the estimation result may be used to update the data recognition model.

At least one of the data learning unit 1210 and the data recognition unit 1220 may be manufactured in the form of at least one hardware chip and mounted in the image processing apparatus. For example, at least one of the data learning unit 1210 and the data recognition unit 1220 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or a conventional general-purpose processor (eg, CPU). Alternatively, it may be manufactured as a part of an application processor) or a graphics-only processor (eg, GPU) and mounted on the various image processing apparatuses described above.

In this case, the data learning unit 1210 and the data recognition unit 1220 may be mounted on one image processing apparatus, or may be mounted on separate image processing apparatuses, respectively. For example, one of the data learner 1210 and the data recognizer 1220 may be included in the image processing apparatus, and the other one may be included in the server. In addition, the data learning unit 1210 and the data recognition unit 1220 may provide the model information built by the data learning unit 1210 to the data recognition unit 1220 through a wired or wireless connection, and the data recognition unit ( Data input to 1220 may be provided to the data learning unit 1210 as additional learning data.

Meanwhile, at least one of the data learning unit 1210 and the data recognition unit 1220 may be implemented as a software module. When at least one of the data learning unit 1210 and the data recognition unit 1220 is implemented as a software module (or a program module including an instruction), the software module is a computer-readable, non-transitory, non-transitory It may be stored in a readable recording medium (non-transitory computer readable media). Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

7 is a block diagram of the data learning unit 1210 according to an exemplary embodiment.

Referring to FIG. 7 , the data learning unit 1210 according to some embodiments includes a data acquiring unit 1210-1, a preprocessing unit 1210-2, a training data selection unit 1210-3, and a model learning unit 1210. -4) and a model evaluation unit 1210-5.

The data acquisition unit 1210-1 is configured to learn the data recognition model (eg, the first data recognition model of FIG. 3 , the second data recognition model of FIG. 3 ) to have a criterion for determining which region in the image corresponds to the information of interest. You can get the data you need. The data acquisition unit 1210-1 may acquire data necessary for learning to determine which region in the image corresponds to the information of interest.

For example, the data acquisition unit 1210-1 may acquire image data, for example, an image or a moving picture. The data acquisition unit 1210-1 may acquire data directly input from the image processing apparatus 1000 or selected data. Also, the data acquisition unit 1210-1 may acquire various types of sensing information sensed by the image processing apparatus 1000 using various sensors. Also, the data acquisition unit 1210-1 may acquire data received from an external device such as the server 2000 that communicates with the image processing apparatus 1000 .

The data acquisition unit 1210-1 may acquire data input from a user, data photographed or previously stored in the image processing apparatus 1000, or data received from an external device such as a server, but is not limited thereto.

The preprocessor 1210 - 2 may preprocess the acquired data so that the acquired data can be used for learning for the data recognition model to determine which region in the image corresponds to the information of interest. The preprocessor 1210-2 may process the acquired data into a preset format so that the model learning unit 1210-4, which will be described later, uses the acquired data for learning for situation determination.

For example, the pre-processing unit 1210-2 removes noise or processes the data such as images and moving images acquired by the data acquisition unit 1210-1 to select meaningful data in a predetermined form. can do.

The training data selector 1210 - 3 may select data required for learning to determine which region in the image corresponds to the interest information among data pre-processed by the data recognition model. The selected data may be provided to the model learning unit 1210 - 4 . The learning data selector 1210-3 is configured for learning to determine which region in the image corresponds to the interest information among the preprocessed data according to a predetermined criterion for determining which region in the image corresponds to the interest information. data can be selected. Also, the training data selection unit 1210 - 3 may select data according to a preset selection criterion by learning by the model learning unit 1210 - 4 to be described later.

The learning data selection unit 1210 - 3 may have a selection criterion for data selection for each data type such as an image or a moving image, and may select data necessary for learning by using the selection criterion.

The learning data selector 1210 - 3 may select data required for learning in order for the data recognition model to learn which region in the image corresponds to the interest information.

The model learning unit 1210 - 4 may learn the data recognition model to have a criterion for determining which region in the image corresponds to the interest information based on the training data. Also, the model learning unit 1210 - 4 may learn a selection criterion for which training data should be used in order for the data recognition model to determine which region in the image corresponds to the information of interest.

The model learning unit 1210 - 4 may learn how the data recognition model determines which region in the image corresponds to the interest information. For example, the model learning unit 1210 - 4 may learn how the first data recognition model determines which region in the image corresponds to the salient region. Also, the model learning unit 1210 - 4 may learn how the second data recognition model determines which region in the image corresponds to the personalized interest information.

Also, the model learning unit 1210 - 4 may learn a data recognition model used to determine which region in the image corresponds to the interest information by using the training data. In this case, the data recognition model may be a pre-built model. For example, the data recognition model may be a model built in advance by receiving basic training data (eg, sample text, etc.).

The model learning unit 1210 - 4 may train the data recognition model through, for example, supervised learning using learning data as an input value. In addition, the model learning unit 1210-4 learns the type of data required to determine which region in the image corresponds to the interest information without any guidance, for example, so that which region in the image corresponds to the interest information. A data recognition model can be trained through unsupervised learning to find a criterion for determining whether to Alternatively, the model learning unit 1210 - 4 may, for example, perform a data recognition model through reinforcement learning using feedback on whether a result of determining which region in an image according to learning corresponds to the information of interest is correct. can be learned

According to an embodiment, the model learning unit 1210-4 receives training data including coordinate information of a region in which at least one of a color or a shape included in an image and a color or shape included in the image is different from that of the first data recognition model. It is possible to learn to have a criterion for estimating the region of interest by using it.

For example, the model learning unit 1210 - 4 may perform learning using a supervised learning method using learning data including an image as shown in FIG. 4A and coordinate information of a region where a human shape included in the image is located. .

For this reason, the first data recognition model learned by the model learning unit 1210 - 4 recognizes an object included in the live view image generated by the image capturing apparatus 1000 , and a peripheral area and color among the recognized objects and/or a region in which an object having a different shape is located may be estimated as a region of interest.

Also, according to an embodiment, the model learning unit 1210 - 4 learns the second data recognition model to have a criterion for estimating the region of interest using the image captured by the user using the image capturing apparatus 1000 . can do.

For example, the model learning unit 1210 - 4 may use an unsupervised learning method to receive an image captured by a user as shown in FIG. 5A and learn to have a criterion for estimating an ROI.

Specifically, the model learning unit 1210 - 4 may recognize an object from an image captured by the user. For example, the model learning unit 1210 - 4 may recognize an object with respect to a plurality of images captured by the user stored in the gallery application. The model learning unit 1210 - 4 may classify objects of similar types into groups according to the types of recognized objects. Alternatively, the model learning unit 1210 - 4 may select the rank of the generated group according to the frequency of the recognized objects. Accordingly, the model learning unit 1210 - 4 may learn by using an object that the user of the image capturing apparatus 1000 mainly shoots and shoots a lot.

For this reason, the second data recognition model learned by the model learning unit 1210 - 4 is an object having a high user preference of the image capturing apparatus 1000 among objects included in the live view image generated by the image capturing apparatus 1000 . A region in which is located may be estimated as a region of interest. That is, the second data recognition model may estimate an ROI tailored to a preference of a user who uses the image capturing apparatus 1000 .

According to various embodiments, the image processing apparatus 1000 may generate a trained fourth data recognition model to estimate only some objects having a high user preference from among the second data recognition models. For example, as a result of learning the image captured by the user, the model learning unit 1210 - 4 may generate group A, group B, group C, and group D. In this case, the image processing apparatus 1000 may generate a fourth data recognition model capable of detecting only an object belonging to group A having the highest frequency among objects detected in an image captured by the user. In this case, the image processing apparatus 1000 may selectively or sequentially use the second data recognition model and the fourth data recognition model according to a given situation.

The data recognition model may be constructed in consideration of the field of application of the recognition model, the purpose of learning, or the computer performance of the device. The data recognition model may be, for example, a model based on a neural network. For example, a model such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent deep neural network (BRDNN) may be used as the data recognition model, but is not limited thereto.

According to various embodiments, when a plurality of pre-built data recognition models exist, the model learning unit 1210 - 4 uses a data recognition model that has a high correlation between the input training data and the basic learning data as a data recognition model to be trained. can decide In this case, the basic learning data may be pre-classified for each type of data, and the data recognition model may be previously built for each type of data. For example, the basic training data is pre-classified by various criteria such as the region where the training data is generated, the time when the training data is generated, the size of the training data, the genre of the training data, the creator of the training data, the type of object in the training data, etc. may have been

Also, the model learning unit 1210 - 4 may train the data recognition model using, for example, a learning algorithm including error back-propagation or gradient descent. .

Also, when the data recognition model is learned, the model learning unit 1210 - 4 may store the learned data recognition model. In this case, the model learning unit 1210 - 4 may store the learned data recognition model in the memory of the image processing apparatus including the data recognition unit 1220 . Alternatively, the model learning unit 1210 - 4 may store the learned data recognition model in the memory of the image processing apparatus including the data recognition unit 1220 to be described later. Alternatively, the model learning unit 1210 - 4 may store the learned data recognition model in a memory of a server connected to the image processing apparatus through a wired or wireless network.

In this case, the memory in which the learned data recognition model is stored may also store, for example, commands or data related to at least one other component of the image processing apparatus. The memory may also store software and/or programs. A program may include, for example, a kernel, middleware, an application programming interface (API) and/or an application program (or "application"), and the like.

The model evaluator 1210-5 may input evaluation data to the data recognition model, and when the recognition result output from the evaluation data does not satisfy a predetermined criterion, the model learning unit 1210-4 may be trained again. have. In this case, the evaluation data may be preset data for evaluating the data recognition model.

For example, the model evaluator 1210-5 sets a predetermined criterion when the number or ratio of evaluation data for which the recognition result is not accurate among the recognition results of the learned data recognition model for the evaluation data exceeds a preset threshold. It can be evaluated as unsatisfactory. For example, when the predetermined criterion is defined as a ratio of 2%, when the learned data recognition model outputs an erroneous recognition result for more than 20 evaluation data out of a total of 1000 evaluation data, the model evaluation unit 1210-5 can be evaluated that the trained data recognition model is not suitable.

On the other hand, when there are a plurality of learned data recognition models, the model evaluation unit 1210 - 5 evaluates whether each learned data recognition model satisfies a predetermined criterion, and recognizes the model that satisfies the predetermined criterion as final data It can be determined as a model. In this case, when there are a plurality of models satisfying the predetermined criteria, the model evaluator 1210 - 5 may determine any one or a predetermined number of models preset in the order of the highest evaluation score as the final data recognition model.

According to various embodiments, the data learning unit 1210 includes a data acquiring unit 1210 - 1 and a model learning unit 1210 - 4 , and a preprocessing unit 1210 - 2 . The training data selection unit 1210 - 3 and the model evaluation unit 1210 - 5 may be selectively included.

On the other hand, in the data learning unit 1210, the data acquisition unit 1210-1, the preprocessor 1210-2, the training data selection unit 1210-3, the model learning unit 1210-4, and the model evaluation unit 1210 At least one of -5) may be manufactured in the form of at least one hardware chip and mounted on the image processing apparatus. For example, at least one of the data acquisition unit 1210-1, the preprocessor 1210-2, the training data selection unit 1210-3, the model learning unit 1210-4, and the model evaluation unit 1210-5 One may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU) as described above. It may be mounted on various image processing apparatuses.

In addition, the data acquisition unit 1210-1, the preprocessor 1210-2, the training data selection unit 1210-3, the model learning unit 1210-4, and the model evaluation unit 1210-5 are one image. It may be mounted on the processing apparatus, or may be respectively mounted on separate image processing apparatuses. For example, some of the data acquisition unit 1210-1, the preprocessor 1210-2, the training data selection unit 1210-3, the model learning unit 1210-4, and the model evaluation unit 1210-5 may be included in the image processing apparatus, and the remaining part may be included in the server.

In addition, at least one of the data acquisition unit 1210-1, the preprocessor 1210-2, the training data selection unit 1210-3, the model learning unit 1210-4, and the model evaluation unit 1210-5 is It may be implemented as a software module. At least one of the data acquisition unit 1210-1, the preprocessor 1210-2, the training data selection unit 1210-3, the model learning unit 1210-4, and the model evaluation unit 1210-5 is a software module When implemented as (or, a program module including instructions), the software module may be stored in a computer-readable non-transitory computer readable medium. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

8 is a block diagram of the data recognition unit 1220 according to an exemplary embodiment.

Referring to FIG. 8 , the data recognition unit 1220 according to some embodiments includes a data acquisition unit 1220-1, a preprocessor 1220-2, a recognition data selection unit 1220-3, and a recognition result providing unit ( 1220-4) and a model update unit 1220-5.

The data recognition unit 1220 uses a data recognition model corresponding to a predetermined condition from among a plurality of learned data recognition models, according to a determination criterion for determining whether the data recognition model corresponds to the learned interest information, which It may be determined whether the region corresponds to the information of interest.

The data acquisition unit 1220-1 may acquire various types of data necessary for the data recognition model to determine which region in the image corresponds to the information of interest. For example, the data acquisition unit 1220-1 may acquire image data such as an image or a moving picture. For example, the data acquisition unit 1220-1 acquires data directly input from the image processing apparatus 1000 or selected data, or obtains various sensing information sensed by the image processing apparatus 1000 using various sensors. can be obtained Also, the data acquisition unit 1220-1 may acquire data received from an external device such as the server 2000 that communicates with the image processing apparatus 1000 .

The preprocessor 1220 - 2 may preprocess the acquired data so that the data recognition model can use the acquired data to determine which region in the image corresponds to the information of interest. The preprocessor 1220-2 processes the acquired data into a preset format so that the recognition result providing unit 1220-4, which will be described later, uses the acquired data to determine which region in the image corresponds to the information of interest. can do.

For example, the preprocessor 1220 - 2 removes noise from the image data such as an image or a moving picture acquired by the data acquisition unit 1220 - 1 to select meaningful data or converts it into a predetermined form. can be processed

The recognition data selector 1220 - 3 may select data necessary for determining which region in the image of the data recognition model corresponds to the information of interest from among the preprocessed data. The selected data may be provided to the recognition result providing unit 1220 - 4 . The recognition data selector 1220 - 3 may select some or all of the pre-processed data according to a predetermined criterion for determining which region in the image corresponds to the information of interest. In addition, the recognition data selection unit 1220-3 may select data according to a preset selection criterion by learning by the model learning unit 1210-4, which will be described later.

The recognition result providing unit 1220 - 4 may apply the selected data to the data recognition model to determine which region in the image corresponds to the information of interest. The recognition result providing unit 1220 - 4 may provide a region corresponding to the interest information in the image according to the purpose of data recognition. The recognition result providing unit 1220 - 4 may apply the selected data to the data recognition model by using the data selected by the recognition data selecting unit 1220 - 3 as an input value. Also, the recognition result may be determined by a data recognition model. The recognition result providing unit 1220 - 4 may determine which region in the image corresponds to the interest information based on a data recognition model corresponding to a predetermined condition among a plurality of data recognition models.

When the first condition is satisfied, the recognition result providing unit 1220-4 may estimate the saliency region as the user's ROI according to a criterion for determining whether the first data recognition model corresponds to the learned saliency region. have. The first data recognition model may use a criterion for determining whether the salient region corresponds to the learned salient region when determining the salient region in the live view image acquired by the photographing unit 1200 .

For example, when the image processing apparatus 1000 satisfies the first condition, the recognition result providing unit 1220-4 detects a phone booth from the input image as shown in FIG. 4B using the first data recognition model, The area in which the phone booth is located may be estimated as the area of interest.

When the second condition is satisfied, the recognition result providing unit 1220-4 selects an area corresponding to the personalized interest information according to a criterion for determining whether the second data recognition model corresponds to the learned personalized interest information. can be estimated as the region of interest of When determining a region corresponding to the personalized interest information in the live view image acquired by the photographing unit 1200 , the second data recognition model may use a criterion for determining whether it corresponds to the learned personalized interest information.

For example, when the image processing apparatus 1000 satisfies the second condition, the recognition result providing unit 1220 - 4 detects the face of a child from the input image as shown in FIG. 5B using the second data recognition model, and , the area with the child's face can be estimated as the area of interest.

In addition, when the model learning unit (eg, the model learning unit 1210 - 4 of FIG. 7 ) learns the image captured by the user, the image including a shape similar to a car has the highest frequency, the recognition result providing unit 1220 -4) may detect a shape similar to a car in the input image as shown in FIG. 5A using the fourth data recognition model, and may estimate a region having a shape similar to a car as a region of interest.

That is, when the user does not satisfy the second condition in the initial purchase of the image processing apparatus 1000 , the image processing apparatus 1000 may estimate the ROI using the first data recognition model. And, when the second condition is satisfied because the number of images captured by the image processing apparatus 1000 increases, the image processing apparatus 1000 determines the region of interest using the second data recognition model and/or the fourth data recognition model. can be estimated

The model updating unit 1220 - 5 may update the data recognition model based on the evaluation of the recognition result provided by the recognition result providing unit 1220 - 4 . For example, the model updating unit 1220-5 provides the model learning unit 1210-4 with a result of determining which region in the image provided by the recognition result providing unit 1220-4 corresponds to the interest information to the model learning unit 1210-4. , the model learning unit 1210 - 4 may update the data recognition model.

According to various embodiments, the data recognition unit 1220 includes a data acquisition unit 1220-1, and a recognition result providing unit 1220-4, a pre-processing unit 1230-2, and a recognition data selection unit 1220. -3), and the model updater 1220 - 5 may be selectively included.

On the other hand, in the data recognition unit 1220, the data acquisition unit 1220-1, the preprocessor 1220-2, the recognition data selection unit 1220-3, the recognition result providing unit 1220-4, and the model update unit ( 1220-5) may be manufactured in the form of at least one hardware chip and mounted on the image processing apparatus. For example, among the data acquisition unit 1220-1, the preprocessor 1220-2, the recognition data selection unit 1220-3, the recognition result providing unit 1220-4, and the model update unit 1220-5 At least one may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU). It may be mounted on one of various image processing apparatuses.

In addition, the data acquisition unit 1220-1, the preprocessor 1220-2, the recognition data selection unit 1220-3, the recognition result providing unit 1220-4, and the model update unit 1220-5 are one unit. It may be mounted on the image processing apparatus, or may be respectively mounted on separate image processing apparatuses. For example, among the data acquisition unit 1220-1, the preprocessor 1220-2, the recognition data selection unit 1220-3, the recognition result providing unit 1220-4, and the model update unit 1220-5 Some may be included in the image processing apparatus, and some may be included in the server.

In addition, at least one of the data acquisition unit 1220-1, the preprocessor 1220-2, the recognition data selection unit 1220-3, the recognition result providing unit 1220-4, and the model update unit 1220-5 may be implemented as a software module. At least one of the data acquisition unit 1220-1, the preprocessor 1220-2, the recognition data selection unit 1220-3, the recognition result providing unit 1220-4, and the model update unit 1220-5 is software. When implemented as a module (or a program module including instructions), the software module may be stored in a computer-readable non-transitory computer readable medium. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

9 is a block diagram illustrating an image processing apparatus according to another exemplary embodiment. Those of ordinary skill in the art related to the present embodiment can see that other general-purpose components other than the components shown in FIG. 9 may be further included. Duplicate descriptions of the components described in FIG. 1 will be omitted.

The external server 2000 classifies the images captured by the user according to a predetermined criterion, learns the third data recognition model from the classified images, and obtains a criterion for determining whether or not it corresponds to personalized interest information. have.

According to an embodiment, the third data recognition model may be trained in a similar manner to the second data recognition model to obtain a criterion for determining interest information tailored to the user.

The image processing apparatus 1000 uses the user's images stored in the external server 2000 to receive, from the external server 2000, a criterion for determining whether the third data recognition model corresponds to the learned personalized interest information. It may further include a communication unit 1500 .

At least one processor constituting the control unit 1200 may determine whether the data recognition model provided in the image processing apparatus 1000 corresponds to the personalized interest information received from the external server 2000, the personalized interest The region corresponding to the information may be estimated as the user's ROI in the live view image.

Meanwhile, the image processing apparatus 1000 may transmit the live view image acquired by the image processing apparatus 1000 to the server 2000 through the communication unit 1500 . The server 2000 selects a region corresponding to the personalized interest information in the live view image transmitted from the image processing apparatus 1000 according to a criterion for determining whether the third data recognition model corresponds to the learned personalized interest information. It may be estimated as the user's ROI, and information on the estimated ROI may be transmitted to the image processing apparatus 1000 .

10 is a diagram illustrating an example of learning and recognizing data by interworking with the image processing apparatus 1000 and the server 2000 according to some embodiments.

Referring to FIG. 10 , the server 2000 may learn a criterion for determining which region in an image corresponds to interest information, and the image processing apparatus 1000 uses the data recognition model learned by the server 2000 . Accordingly, it is possible to determine which region in the image corresponds to the interest information.

In this case, the data learning unit 2210 of the server 2000 may perform the function of the data learning unit 1210 illustrated in FIG. 7 . The data learning unit 2210 of the server 2000 relates to which data to use to determine which region in the image corresponds to the interest information, and how to determine which region in the image corresponds to the interest information using the data. Judgment criteria can be learned. The data learning unit 2210 of the server 2000 acquires data to be used for learning, and applies the acquired data to a data recognition model to be described later, thereby determining which region in the image corresponds to the information of interest. can learn

In addition, the recognition result providing unit 1220 - 4 of the image processing apparatus 1000 applies the data selected by the recognition data selection unit 1220 - 3 to the data recognition model generated by the server 2000 to apply any It may be determined whether the region corresponds to the information of interest. For example, the recognition result providing unit 1220-4 transmits the data selected by the recognition data selection unit 1220-3 to the server 2000, and the server 2000 performs the recognition data selection unit 1220-3 It may be requested to determine which region in the image corresponds to the information of interest by applying the data selected by the to the data recognition model. Also, the recognition result providing unit 1220 - 4 may receive a region corresponding to the interest information in the image determined by the server 2000 from the server 2000 .

For example, the image processing apparatus 1000 may transmit a live view image acquired by the image processing apparatus 1000 to the server 2000 . The server 2000 may determine which region in the image corresponds to the interest information by applying the live view image received from the image processing apparatus 1000 to the data recognition model stored in the server 2000 . The server 2000 may further reflect the user's images stored in the server 2000 to determine which region in the image corresponds to the interest information. The region corresponding to the interest information in the image determined by the server 2000 may be transmitted to the image processing apparatus 1000 .

Alternatively, the recognition result providing unit 1320 - 4 of the image processing apparatus 1000 receives the data recognition model generated by the server 2000 from the server 2000 and uses the received data recognition model to select any one in the image. It may be determined whether the region corresponds to the information of interest. In this case, the recognition result providing unit 1220 - 4 of the image processing apparatus 1000 applies the data selected by the recognition data selector 1220 - 3 to the data recognition model received from the server 2000 to apply any It may be determined whether the region corresponds to the information of interest.

For example, the image processing apparatus 1000 may determine which region in the image corresponds to the interest information by applying the live view image acquired by the image processing apparatus 1000 to the data recognition model received from the server 2000 . can The server 2000 may transmit the user's images stored in the server 2000 to the image processing apparatus 1000 so that the image processing apparatus 1000 can further use it when determining which region in the image corresponds to the information of interest. have.

11 is a block diagram illustrating an image processing apparatus 1000 according to another exemplary embodiment.

11 , the image processing apparatus 1000 according to another exemplary embodiment includes a memory 1100 , a controller 1200 , an input/output unit 1300 , a sensing unit 1400 , a communication unit 1500 , and an A/ A V input unit 1600 may be included.

The memory 1100 may store a program for processing and control by the controller 1200 , and may also store data input to or output from the image processing apparatus 1000 . Memory 1100 may store computer executable instructions.

The memory 1100 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory), and a RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , may include at least one type of storage medium among optical disks.

Programs stored in the memory 1100 may be classified into a plurality of modules according to their functions, for example, may be classified into a UI module, a touch screen module, a notification module, and the like.

The UI module may provide a specialized UI, GUI, or the like that is interlocked with the image processing apparatus 1000 for each application. The touch screen module may detect a touch gesture on the user's touch screen and transmit information about the touch gesture to the controller 1200 . The touch screen module according to some embodiments may recognize and analyze a touch code. The touch screen module may also be configured as separate hardware. The user's touch gesture may include tap, touch & hold, double tap, drag, pan, flick, drag and drop, swipe, and the like. The notification module may generate a signal for notifying the occurrence of an event in the image processing apparatus 1000 . Examples of events occurring in the image processing apparatus 1000 include message reception, key signal input, content input, content transmission, and content detection corresponding to a predetermined condition. The notification module may output a notification signal in the form of a video signal through the display unit 1322 , may output a notification signal in the form of an audio signal through the sound output unit 1324 , and vibrate through the vibration motor 1326 . A notification signal may be output in the form of a signal.

The controller 1200 generally controls the overall operation of the image processing apparatus 1000 . For example, the controller 1200 controls the input/output unit 1300 , the sensing unit 1400 , the communication unit 1500 , the A/V input unit 1600 , etc. in general by executing programs stored in the memory 1100 . can do.

Specifically, the controller 1200 may include at least one processor. The control unit 1200 may include a plurality of processors or a single processor in an integrated form according to its function and role.

At least one processor constituting the control unit 1200 executes computer-executable instructions stored in the memory 1100 so that the data recognition model learns, based on a data recognition model corresponding to a predetermined condition among the data recognition models. The user's ROI may be estimated from the acquired live-view image according to a criterion for determining whether the information corresponds to the ROI, and the estimated ROI may be focused.

When the first condition is satisfied, the at least one processor constituting the control unit 1200 estimates the saliency region as the user's ROI according to a criterion for determining whether the first data recognition model corresponds to the learned saliency region. and, when the second condition is satisfied, the region corresponding to the personalized interest information may be estimated as the user's interest region according to a criterion for determining whether the second data recognition model corresponds to the learned personalized interest information. .

The learned data recognition model may be stored in a server external to the image processing apparatus 1000 , and may be received from the server according to a request of the image processing apparatus 1000 .

The input/output unit 1300 may include a user input unit 1310 and an output unit 1320 . The input/output unit 1300 may be in a form in which the user input unit 1310 and the output unit 1320 are separated, or may be in one integrated form such as a touch screen.

The input/output unit 1300 may display a live view image focused on the estimated ROI.

The user input unit 1310 may mean a means for a user to input data for controlling the image processing apparatus 1000 . The user input unit 1310 may receive an abbreviation input from the user, and may receive a selection of a sentence corresponding to the abbreviation word from the user.

The user input unit 1310 includes a keypad 1312 , a touch panel 1314 (contact capacitive method, pressure resistance film method, infrared sensing method, surface ultrasonic conduction method, integral tension measurement method, piezoelectric type). effect method, etc.), a fan recognition panel 1316, and the like. In addition, the user input unit 1310 may include a jog wheel, a jog switch, and the like, but is not limited thereto.

The output unit 1320 may output a result of executing an application in the image processing apparatus 1000 . The output unit 1320 may output an operation result of the image processing apparatus 1000 , and when there is a user input, may output a result changed according to the user's input.

The output unit 1320 may output an audio signal, a video signal, or a vibration signal, and the output unit 1320 may include a display unit 1322 , a sound output unit 1324 , and a vibration motor 1326 . have.

The display 1322 displays information processed by the image processing apparatus 1000 . For example, the display 1322 may display an execution screen of a camera application or display a user interface for receiving a user's manipulation.

On the other hand, when the display unit 1322 and the touch pad form a layer structure to form a touch screen, the display unit 1322 may be used as an input device in addition to an output device. The display unit 1322 includes a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a three-dimensional display ( 3D display) and electrophoretic display (electrophoretic display) may include at least one. Also, depending on the implementation form of the image processing apparatus 1000 , the image processing apparatus 1000 may include two or more display units 1322 . In this case, two or more display units 1322 may be disposed to face each other using a hinge.

The sound output unit 1324 outputs audio data received from the communication unit 1500 or stored in the memory 1100 . Also, the sound output unit 1324 outputs a sound signal related to a function (eg, a call signal reception sound, a message reception sound, and a notification sound) performed by the image processing apparatus 1000 . The sound output unit 1324 may include a speaker, a buzzer, and the like.

The vibration motor 1326 may output a vibration signal. For example, the vibration motor 1326 may output a vibration signal corresponding to the output of audio data or video data (eg, a call signal reception sound, a message reception sound, etc.). Also, the vibration motor 1326 may output a vibration signal when a touch is input to the touch screen.

The sensing unit 1400 may detect a state of the image processing apparatus 1000 or a state around the image processing apparatus 1000 , and transmit the sensed information to the controller 1200 .

The sensing unit 1400 includes a magnetic sensor 1410 , an acceleration sensor 1420 , a temperature/humidity sensor 1430 , an infrared sensor 1440 , a gyroscope sensor 1450 , and a position sensor. (eg, GPS) 1460 , a barometric pressure sensor 1470 , a proximity sensor 1480 , and at least one of an illuminance sensor 1490 , but is not limited thereto. Since a function of each sensor can be intuitively inferred from the name of a person skilled in the art, a detailed description thereof will be omitted.

The communication unit 1500 may include one or more components that enable communication between the image processing apparatus 1000 and another device or server. For example, the communication unit 1500 may include a short-distance communication unit 1510 , a mobile communication unit 1520 , and a broadcast receiving unit 1530 .

Short-range wireless communication unit (151), Bluetooth communication unit, BLE (Bluetooth Low Energy) communication unit, short-range wireless communication unit (Near Field Communication unit), WLAN (Wi-Fi) communication unit, Zigbee (Zigbee) communication unit, infrared ( It may include an IrDA, infrared Data Association) communication unit, a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, an Ant+ communication unit, and the like, but is not limited thereto.

The mobile communication unit 1520 transmits/receives a radio signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call signal, or a text/multimedia message.

The broadcast receiver 1530 receives a broadcast signal and/or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. According to an embodiment, the image processing apparatus 1000 may not include the broadcast receiver 1530 .

Also, the communication unit 1500 may communicate with other devices, servers, peripheral devices, and the like in order to transmit/receive or upload content.

The A/V (Audio/Video) input unit 1600 is for inputting an audio signal or a video signal, and may include a photographing unit 1610 , a microphone 1620 , and the like. The photographing unit 1610 may obtain an image frame such as a still image or a moving image through an image sensor in a video call mode or a photographing mode. The image captured through the image sensor may be processed through the controller 1200 or a separate image processing unit (not shown).

The image frame processed by the photographing unit 1610 may be stored in the memory 1100 or transmitted to the outside through the communication unit 1500 . Two or more photographing units 1610 may be provided according to the configuration of the terminal.

The microphone 1620 receives an external sound signal and processes it as electrical voice data. For example, the microphone 1620 may receive an acoustic signal from an external device or a speaker. The microphone 1620 may use various noise removal algorithms for removing noise generated in the process of receiving an external sound signal.

12 is a flowchart illustrating an image processing method according to an exemplary embodiment.

In operation 1210 , the image processing apparatus 1000 acquires a live view image including at least one subject.

In operation 1220 , the image processing apparatus 1000 determines whether the data recognition model corresponds to the learned interest information based on a data recognition model corresponding to a predetermined condition among the data recognition models, according to the obtained live view. Estimate the user's ROI from the image.

When the first condition is satisfied, the image processing apparatus 1000 estimates the saliency region as the user's ROI according to a criterion for determining whether the first data recognition model corresponds to the learned saliency region, and the second condition is satisfied, the region corresponding to the personalized interest information may be estimated as the user's interest region according to a criterion for determining whether the second data recognition model corresponds to the learned personalized interest information. The protrusion area is determined by a predetermined criterion regarding the area occupied by the subject in the image or the color distribution of the image, and the personalized interest information relates to user images stored in the image processing apparatus 1000 in which the image processing method is performed. It may be determined by predetermined statistics. In this case, the second condition is that the number of images stored in the image processing apparatus 1000 on which the image processing method is performed is greater than the predetermined number, and the reliability of the personalized interest information satisfies the predetermined condition, and the first condition The condition means that the second condition is not satisfied.

Meanwhile, the image processing apparatus 1000 classifies the user's images according to a predetermined criterion, trains a third data recognition model with the classified images, and obtains a criterion for determining whether the user's image corresponds to the personalized interest information. Communication with the server 2000 may be performed. The image processing apparatus 1000 may use the user's images stored in the external server 2000 to receive, from the external server 2000, a criterion for determining whether the third data recognition model corresponds to the learned personalized interest information. have. According to a criterion for determining whether the data recognition model provided in the image processing apparatus 1000 corresponds to the received personalized interest information, a region corresponding to the personalized interest information may be estimated as the user's region of interest.

The image processing apparatus 1000 may estimate a region corresponding to the interest information having a high priority as the user's region of interest based on the priority of the interest information. When a plurality of ROIs are estimated, the image processing apparatus 1000 may perform multi-focusing by focusing on all of the plurality of ROIs. When a plurality of regions of interest are estimated, the image processing apparatus 1000 may focus on a region of interest selected by the user from among the plurality of regions of interest.

In operation 1230 , the image processing apparatus 1000 focuses on the estimated ROI.

In operation 1240 , the image processing apparatus 1000 displays a live view image focused on the estimated ROI. The image processing apparatus 1000 may capture an image by focusing on the estimated ROI in response to a user's photographing command.

13 is a flowchart illustrating a situation of estimating a region of interest when an image processing apparatus includes a first processor and a second processor according to an exemplary embodiment.

The first processor 1200a controls execution of at least one application installed in the image processing apparatus 1000 , and provides graphics for images (eg, live view images, captured images, etc.) acquired by the image processing apparatus 1000 . processing can be performed. The first processor 1200a may be implemented in the form of a system on chip (SoC) in which functions such as a central processing unit (CPU), a graphic processing unit (GPU), a communication chip, and a sensor are integrated. Also, the first processor 1200a may be referred to as an application processor (AP) in this specification.

The second processor 1200b may estimate the ROI of the image using the data recognition model.

Meanwhile, the second processor 1200b may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI) that performs a function of estimating a region of interest using a data recognition model. According to various embodiments, in the case of a data recognition model using visual understanding as an element technology, a dedicated hardware chip for artificial intelligence (AI) may include a GPU.

According to an embodiment, the image processing apparatus 1000 may include a first processor 1200a and a second processor 1200b.

In operation S1310, the first processor 1200a may acquire a live view image. The live view image may include, for example, at least one subject.

In operation S1320 , the first processor 1200a may determine a data recognition model corresponding to a predetermined condition.

For example, when the image processing apparatus 1000 satisfies the second condition described with reference to FIG. 3 , the first processor 1200a may determine to use the second data recognition model or the fourth data recognition model. According to an embodiment, the first processor 1200a may select one of the second data recognition model and the fourth data recognition model according to a default value set by the manufacturer of the image processing apparatus 1000 or a user's selection.

Also, when the image processing apparatus 1000 does not satisfy the second condition, the first processor 1200a may determine that the first condition is satisfied and determine to use the first data recognition model.

In operation S1330 , the first processor 1200a may request the second processor 1200b to estimate the ROI using the determined data recognition model.

In operation S1340 , the second processor 1200b may estimate the ROI using the determined data recognition model.

For example, when the first data recognition model is determined, the second processor 1200b sets the saliency region as a region of interest of the user according to a criterion for determining whether the first data recognition model corresponds to the learned saliency region. can be estimated When the second data recognition model is determined, the second processor 1200b selects an area corresponding to the personalized interest information of the user according to a criterion for determining whether the second data recognition model corresponds to the learned personalized interest information. It can be estimated as a region of interest.

When the fourth data recognition model is determined, the second processor 1200b may estimate the region corresponding to the personalized Guam SIM information as the user's ROI by using the fourth data recognition model.

In operation S1350 , the second processor 1200b may transmit the estimated ROI to the first processor 1200a.

In operation S1360 , the first processor 1200a may focus on the estimated ROI.

In operation S1370 , the first processor 1200a may display a live view image focused on the estimated ROI.

14 is a flowchart illustrating a situation in which an ROI is estimated when an image processing apparatus includes a first processor, a second processor, and a third processor, according to an exemplary embodiment.

According to an embodiment, the image processing apparatus 1000 may include a first processor 1200a, a second processor 1200b, and a third processor 1200c. For example, the second processor 1200b may estimate the ROI using the first data recognition model. The third processor 1200c may estimate the ROI by using the second data recognition model or the fourth data recognition model.

In operation S1410, the first processor 1200a may acquire a live view image. The live view image may include, for example, at least one subject.

In operation S1420, the first processor 1200a may determine a data recognition model corresponding to a predetermined condition.

For example, when the image processing apparatus 1000 satisfies the second condition described with reference to FIG. 3 , the first processor 1200a may determine to use the second data recognition model or the fourth data recognition model.

According to an embodiment, the first processor 1200a may set the fourth data recognition model as a basic data recognition model corresponding to the second condition in order to increase the speed of detecting the user's ROI. However, the present invention is not limited thereto. For example, the first processor 1200a may select one of the second data recognition model and the fourth data recognition model according to a default value set by the manufacturer of the image processing apparatus 1000 or a user's selection.

Also, when the image processing apparatus 1000 does not satisfy the second condition, the first processor 1200a may determine that the first condition is satisfied and determine to use the first data recognition model.

Referring to step S1430 , when the first data recognition model is determined, the first processor 1200a may request the second processor 1200b to estimate the ROI. Also, when the second data recognition model or the fourth data recognition model is determined, the first processor 1200a may request the third processor 1200c to estimate the ROI.

In operation S1440 , the third processor 1200c may estimate the ROI using the fourth data recognition model.

In operation S1450 , the third processor 1200c may check whether the estimation of the ROI is completed using the fourth data recognition model.

In operation S1470 , when it is confirmed that the ROI is estimated using the fourth data recognition model, the third processor 1200c may transmit the estimated ROI to the first processor 1200a.

In operation S1460 , the second processor 1200b may estimate the ROI using the first data recognition model. Also, if it is confirmed in step S1450 that the third processor 1200c has failed to estimate the region of interest using the fourth data recognition model, the second processor 1200b estimates the region of interest using the first data recognition model. can do.

In operation S1475 , the second processor 1200b may transmit the ROI estimated using the first data recognition model to the first processor 1200a.

In operation S1480 , the first processor 1200a may focus on the estimated ROI.

In operation S1490 , the first processor 1200a may display a live view image focused on the estimated ROI.

15 is a flowchart illustrating another situation in which an ROI is estimated when an electronic device includes a first processor, a second processor, and a third processor, according to an embodiment.

In operation S1510, the first processor 1200a may acquire a live view image. The live view image may include, for example, at least one subject.

In operation S1520, the first processor 1200a may determine a data recognition model corresponding to a predetermined condition.

For example, when the image processing apparatus 1000 satisfies the second condition described with reference to FIG. 3 , the first processor 1200a may determine to use the second data recognition model or the fourth data recognition model.

According to an embodiment, the first processor 1200a may set the fourth data recognition model as a basic data recognition model corresponding to the second condition in order to increase the speed of detecting the user's ROI. However, the present invention is not limited thereto. For example, the first processor 1200a may select one of the second data recognition model and the fourth data recognition model according to a default value set by the manufacturer of the image processing apparatus 1000 or a user's selection.

Also, when the image processing apparatus 1000 does not satisfy the second condition, the first processor 1200a may determine that the first condition is satisfied and determine to use the first data recognition model.

Referring to step S1530 , when the first data recognition model is determined, the first processor 1200a may request the second processor 1200b to estimate the ROI. Also, when the second data recognition model or the fourth data recognition model is determined, the first processor 1200a may request the third processor 1200c to estimate the ROI.

In operation S1540 , the second processor 1200b may estimate the ROI using the first data recognition model.

In operation S1560 , the second processor 1200b may transmit the estimated ROI to the first processor 1200a.

In operation S1550 , the third processor 1200c may estimate the ROI using the fourth data recognition model.

In operation S1570 , the third processor 1200c may check whether the estimation of the ROI is completed using the fourth data recognition model.

In step S1565. When it is confirmed that the ROI is estimated using the fourth data recognition model, the third processor 1200c may transmit the estimated ROI to the first processor 1200a.

Referring to step S1580 , if it is confirmed that the initialization of the ROI has failed using the fourth data recognition model, the third processor 1200c may estimate the ROI using the second data recognition model.

For example, the fourth data recognition model may estimate a car-like shape as a region of interest. When the object detected from the live-view image does not have a car-like shape, the third processor 1200c performs the second data Another object customized to the user can be estimated as the region of interest by applying the recognition model.

In operation S1590 , the second processor 1200b may transmit the ROI estimated using the first data recognition model to the first processor 1200a.

In operation S1593 , the first processor 1200a may focus on the estimated ROI.

In operation S1595 , the first processor 1200a may display a live view image focused on the estimated ROI.

16 is a flowchart illustrating a situation in which an image processing apparatus estimates a region of interest using a server, according to an exemplary embodiment.

In this case, an interface for transmitting/receiving data between the image processing apparatus 1000 and the server 2000 may be defined.

For example, an API (application program interface) having training data to be applied to the data recognition model as a factor value (or a parameter value or a transfer value) may be defined. The API may be defined by any one protocol (eg, image processing It can be defined as a set of subroutines or functions that can be called for some processing of another protocol (eg, the protocol defined in the server 2000) in the device 1000). An environment in which an operation of another protocol can be performed in one protocol can be provided through .

According to an embodiment, the server 2000 may include a third data recognition model.

In operation S1610 , the image processing apparatus 1000 may acquire a live view image. The live view image may include, for example, at least one subject.

In operation S1620, the image processing apparatus 1000 may determine a data recognition model corresponding to a predetermined condition.

For example, when the second condition described above with reference to FIG. 3 is satisfied, the image processing apparatus 1000 may determine to use the second data recognition model or the fourth data recognition model. Also, when the second condition is not satisfied, the image processing apparatus 1000 may determine that the first condition is satisfied and determine to use the first data recognition model.

Referring to step S1630 , the image processing apparatus 1000 may determine whether the ROI is estimated using the data recognition model.

Referring to operation S1640 , when the ROI is estimated using the data recognition model included in the image processing apparatus 1000 , the image processing apparatus 1000 may focus on the estimated ROI.

Referring to step S1650 , when the ROI is not estimated using the data recognition model included in the image processing apparatus 1000 , the image processing apparatus 1000 may request the server 2000 to estimate the ROI.

In operation S1660, the server 2000 may estimate the ROI using the third data recognition model.

In operation S1670 , the server 2000 may transmit the estimated ROI to the image processing apparatus 1000 .

In operation S1680 , the image processing apparatus 1000 may focus on the estimated ROI.

In operation S1690, the image processing apparatus 1000 may display a live view image focused on the estimated ROI.

Meanwhile, the above-described image processing method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates such a program using a computer-readable storage medium. These computer-readable storage media include read-only memory (ROM), random-access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, and DVDs. -Storing ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetic tapes, floppy disks, magneto-optical data devices, optical data storage devices, hard disks, solid-state disks (SSDs), and may store instructions or software, related data, data files, and data structures, and are stored on a processor or computer so that the processor or computer may execute the instructions. It can be any device capable of providing instructions or software, associated data, data files, and data structures.

In addition, the disclosed embodiments may be implemented as a S/W program including instructions stored in a computer-readable storage medium.

The computer is an apparatus capable of calling a stored instruction from a storage medium and operating according to the called instruction according to the disclosed embodiment, and may include the image processing apparatus according to the disclosed embodiment.

The computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not include a signal and is tangible, and does not distinguish that data is semi-permanently or temporarily stored in the storage medium.

In addition, the control method according to the disclosed embodiments may be provided included in a computer program product (computer program product). Computer program products may be traded between sellers and buyers as commodities.

The computer program product may include a S/W program and a computer-readable storage medium in which the S/W program is stored. For example, computer program products may include products (eg, downloadable apps) in the form of S/W programs distributed electronically through manufacturers of image processing devices or electronic markets (eg, Google Play Store, App Store). can For electronic distribution, at least a portion of the S/W program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing a SW program.

The computer program product, in a system including a server and an image processing device, may include a storage medium of the server or a storage medium of the image processing device. Alternatively, when there is a third device (eg, a smartphone) that is communicatively connected to the server or the image processing device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S/W program itself transmitted from the server to the image processing device or the third device, or transmitted from the third device to the image processing device.

In this case, one of the server, the image processing apparatus, and the third apparatus may execute a computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the image processing device, and the third device may execute a computer program product to distribute the method according to the disclosed embodiments.

For example, a server (eg, a cloud server or an artificial intelligence server) may execute a computer program product stored in the server to control the image processing apparatus communicatively connected to the server to perform the method according to the disclosed embodiments.

As another example, the third device may execute a computer program product to control the image processing device communicatively connected to the third device to perform the method according to the disclosed embodiment. When the third device executes the computer program product, the third device may download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may execute the computer program product provided in a preloaded state to perform the method according to the disclosed embodiments.

So far, examples have been mainly looked at. Those of ordinary skill in the art to which the disclosed embodiments pertain will understand that the disclosed embodiments can be implemented in modified forms without departing from the essential characteristics. Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the invention is indicated in the claims rather than the description of the above-described embodiments, and all differences within an equivalent range should be construed as being included in the scope of the invention.

Claims (20)

a photographing unit configured to acquire a live view image including at least one subject;
a memory storing computer executable instructions;
By executing the computer-executable instruction, based on a data recognition model corresponding to a predetermined condition among data recognition models, the obtained live view is based on a criterion for determining whether the data recognition model corresponds to the learned interest information. at least one processor for estimating a user's ROI from the image and focusing on the estimated ROI; and
an input/output unit configured to display the live view image focused on the estimated ROI;
includes,
the at least one processor,
When the first condition is satisfied, according to a criterion for determining whether the first data recognition model corresponds to the learned saliency area, the saliency area is estimated as the user's area of interest, and when the second condition is satisfied, the second condition 2 The image processing apparatus for estimating the region corresponding to the personalized interest information as the user's region of interest according to a criterion for determining whether the data recognition model corresponds to the learned personalized interest information.

delete The method of claim 1,
The protrusion area is determined by a predetermined criterion regarding an area occupied by a subject in an image or a color distribution of the image, and the personalized interest information is stored in the image processing device according to predetermined statistics on the user's images. To be determined, an image processing device.
The method of claim 1,
The second condition is a case in which the number of images stored in the image processing apparatus is greater than a predetermined number, the reliability of the personalized interest information satisfies a predetermined condition, and the first condition is that the second condition is satisfied. If it is not satisfied, the image processing device.
The method of claim 1,
Using the images of the user stored in the external server, further comprising a communication unit for receiving from the external server a criterion for determining whether the third data recognition model corresponds to the learned personalized interest information,
The at least one processor,
An image processing apparatus for estimating a region corresponding to the personalized interest information as the user's region of interest according to a criterion for determining whether a data recognition model provided in the image processing apparatus corresponds to the received personalized interest information .
6. The method of claim 5,
The external server classifies the user's images according to a predetermined criterion, learns the third data recognition model from the classified images, and obtains a criterion for determining whether the image corresponds to the personalized interest information. processing unit.
The method of claim 1,
The at least one processor,
An image processing apparatus for estimating a region corresponding to the interest information having a higher priority as the user's region of interest, based on the priority of the interest information.
The method of claim 1,
the at least one processor,
When a plurality of ROIs are estimated, the image processing apparatus performs multi-focus for focusing on all of the plurality of ROIs.
The method of claim 1,
The at least one processor,
When a plurality of regions of interest are estimated, the image processing apparatus focuses on a region of interest selected by a user from among the plurality of regions of interest.
The method of claim 1,
The image processing apparatus, wherein the photographing unit captures an image by focusing on the estimated ROI in response to the user's photographing command.
acquiring a live view image including at least one subject;
Based on a data recognition model corresponding to a predetermined condition among data recognition models, according to a criterion for determining whether the data recognition model corresponds to the learned interest information, the user's region of interest is estimated from the acquired live view image step;
focusing on the estimated ROI; and
displaying the live view image focused on the estimated ROI;
includes,
The step of estimating the region of interest comprises:
When the first condition is satisfied, according to a criterion for determining whether the first data recognition model corresponds to the learned saliency area, the saliency area is estimated as the user's area of interest, and when the second condition is satisfied, the second condition 2 An image processing method of estimating a region corresponding to the personalized interest information as the user's region of interest according to a criterion for determining whether the data recognition model corresponds to the learned personalized interest information.
delete 12. The method of claim 11,
The protrusion region is determined by a predetermined criterion regarding the area occupied by the subject in the image or the color distribution of the image, and the personalized interest information is stored in the user's images stored in the image processing device on which the image processing method is performed. An image processing method that is determined by predetermined statistics regarding
12. The method of claim 11,
The second condition is a case in which the number of images stored in the image processing apparatus on which the image processing method is performed is greater than a predetermined number, the reliability of the personalized interest information satisfies a predetermined condition, and the first condition is a case where the second condition is not satisfied, the image processing method.
15. The method of claim 14,
Using the user's images stored in the external server, further comprising the step of receiving, from the external server, a criterion for determining whether a third data recognition model corresponds to the learned personalized interest information,
The step of estimating the region of interest comprises:
According to a criterion for determining whether a data recognition model provided in an image processing apparatus on which the image processing method is performed corresponds to the received personalized interest information, the region corresponding to the personalized interest information is defined as the user's region of interest. Estimating, image processing method.
16. The method of claim 15,
The external server classifies the user's images according to a predetermined criterion, learns the third data recognition model from the classified images, and obtains a criterion for determining whether the image corresponds to the personalized interest information. processing method.
12. The method of claim 11,
The step of estimating the region of interest comprises:
An image processing method of estimating, as a user's region of interest, a region corresponding to the interest information having a higher priority, based on the priority of the interest information.
12. The method of claim 11,
The focusing step is
When a plurality of ROIs are estimated, multi-focusing is performed to focus on all of the plurality of ROIs.
12. The method of claim 11,
The focusing step is
When a plurality of regions of interest are estimated, an image processing method of focusing on a region of interest selected by a user from among the plurality of regions of interest.
acquiring a live view image including at least one subject;
Based on a data recognition model corresponding to a predetermined condition among data recognition models, according to a criterion for determining whether the data recognition model corresponds to the learned interest information, the user's region of interest is estimated from the acquired live view image step;
focusing on the estimated ROI; and
displaying the live view image focused on the estimated ROI; includes,
The step of estimating the region of interest comprises:
When the first condition is satisfied, according to a criterion for determining whether the first data recognition model corresponds to the learned saliency area, the saliency area is estimated as the user's area of interest, and when the second condition is satisfied, the second condition 2 According to a criterion for determining whether the data recognition model corresponds to the learned personalized interest information, instructions set to perform an image processing method of estimating the region corresponding to the personalized interest information as the user's region of interest are stored A computer-readable recording medium.

Images