KR102239015B1 - Image alignment method and apparatus thereof - Google Patents

Abstract

A method of automatically performing image alignment without user input is provided. An image alignment method according to an embodiment of the present invention performed by an image alignment device includes recognizing at least one person from an input image, determining a person of interest among the recognized people, and the input image Performing image alignment based on a person of interest, and performing the image alignment without a user input of the image alignment device for image alignment.

Description

TECHNICAL FIELD [0001] Image alignment method and apparatus thereof

The present invention relates to an image alignment method and apparatus. In more detail, the present invention relates to a method of determining a person of interest among at least one person recognized in an input image, and performing automatic image alignment on the input image based on the person of interest, and an apparatus for performing the method.

As image processing technology is advanced, smart devices or intelligent systems based on image processing technology are being built and used in various fields. For example, an intelligent video surveillance system that detects an object of interest in a surveillance image and detects various events using image processing technology has been constructed and is being used in the surveillance field of various environments. For another example, a smart mirror that photographs surrounding images using a camera attached to the front side and provides various contents based on an analysis result of the captured image is installed and used in various places.

A problem common to the above-described smart devices and intelligent systems is that an automatic alignment function based on an object of interest is not provided. For example, in an intelligent image analysis system, in order for the administrator to check the surveillance image focused on the object of interest, an object of interest is designated from an original image captured by CCTV (closed circuit television), and an image containing the specified object of interest. You have to go through a series of processes, such as zooming. Since the above-described series of processes must be performed directly by the administrator, user convenience is deteriorated and the system cannot be effectively utilized. For another example, in a smart mirror, when a plurality of people are located in the front, the smart mirror must determine any one of the plurality of people as a person of interest and provide related content. In such a case, when a plurality of people provide content for a person of interest along with a captured image, the satisfaction level of the user who is provided with the service is inevitably lowered.

Accordingly, in order to solve the above-described problem, there is a need for a method of determining an object of interest from an input image and performing automatic image alignment based on the object of interest in consideration of various criteria determined according to the purpose of an apparatus and a system.

Korean Patent Publication No. 10-2007-0087085 (published on August 29, 2007)

The technical problem to be solved by the present invention is to provide a method for performing automatic image alignment on an input image based on an object of interest, and an apparatus for performing the method.

Another technical problem to be solved by the present invention is to provide a method of determining an object of interest among at least one object included in an input image and an apparatus for performing the method.

Another technical problem to be solved by the present invention is to provide a method of determining a person of interest among at least one person included in an input image, and configuring a screen for providing a customized service for the person of interest, and an apparatus for performing the method. It is to do.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above technical problem, an image alignment method according to an embodiment of the present invention includes, by an image alignment device, recognizing at least one person from an input image, and the image alignment device includes: Determining a person, and performing, by the image alignment apparatus, image alignment with respect to the input image based on the person of interest, but performing the image alignment without a user input of the image alignment device for the image alignment. It may include.

In one embodiment, determining the person of interest, determining the person of interest, analyzing the demographic characteristics of each of the recognized persons, and using the analyzed demographic characteristics, the It may include the step of determining a person of interest, among the recognized persons, having characteristics corresponding to predetermined demographic information.

In an embodiment, the method may include extracting a face region for each of the recognized persons, and determining a person having the largest extracted face region among the recognized persons as a person of interest.

In one embodiment, the determining of the person of interest includes detecting an object having a designated shape at a designated location of each recognized person, and detecting an object having the designated shape at a designated location of the first person. In this case, it may include determining the first person as the person of interest.

In one embodiment, the determining of the person of interest includes detecting an object having a toothbrush shape at a position around the mouth of each recognized person, and detecting an object having a toothbrush shape at a position around the mouth of the first person When detected, determining the first person as the person of interest may be included.

In one embodiment, the determining of the person of interest includes recognizing a voice and when a text corresponding to the recognized voice is detected in an area inside the outline of the first person, the first person is designated as the person of interest. It may include the step of determining.

In one embodiment, determining the person of interest may include determining a person of interest corresponding to a current situation through a context-based person of interest determination logic for each of the recognized persons. have.

An image alignment apparatus according to another embodiment of the present invention for solving the above-described technical problem includes at least one processor, a network interface, a memory for loading a computer program executed by the processor, and a memory for storing the computer program. Including a storage, the computer program, an operation of recognizing at least one person from an input image, an operation of determining a person of interest among the recognized persons, and image alignment with respect to the input image based on the person of interest, And an operation of performing the image alignment without a user's input of the image alignment device for the image alignment.

A computer program according to another embodiment of the present invention for solving the above-described technical problem, in combination with a computing device, recognizing at least one person from an input image, determining a person of interest among the recognized persons And performing image alignment with respect to the input image based on the person of interest, but may be stored in a recording medium to execute the step of performing the image alignment without a user input for the image alignment.

A smart mirror according to another embodiment of the present invention for solving the above-described technical problem, a front camera sensor that generates an input image, recognizes at least one person from the input image, and selects a person of interest among the recognized people. An image alignment processor that performs image alignment with respect to the input image based on the person of interest, and performs the image alignment without a user input for the image alignment, and a display that displays a result image of the image alignment It may include.

A method of configuring a user-targeted service providing screen according to another embodiment of the present invention for solving the above-described technical problem includes, by a service providing device, recognizing at least one person from an input image, and the service providing device , Determining a person of interest among the recognized persons, and the service providing device constructs and displays a service providing screen targeting the interested person with respect to the input image, but without a user input for configuring the service providing screen. It may include the step of configuring and displaying the service providing screen.

A method of configuring a user-targeted service providing screen according to another embodiment of the present invention for solving the above-described technical problem includes, by a service providing device, recognizing at least one person from an input image, and the service providing device , Receiving biometric data from a biometric signal sensor of the recognized person, determining, by the service providing device, a person of interest among the recognized person by using the received biometric data, and the service providing device, Configuring and displaying a service providing screen for the interested person with respect to the input image, and configuring and displaying the service providing screen without a user input for configuring the service providing screen.

According to the present invention described above, a person of interest may be determined from among at least one person recognized in an input image according to a predetermined criterion, and image alignment may be automatically performed for the person of interest. The automatic image alignment as described above may be used in various fields to improve user immersion and convenience.

For example, when automatic image alignment is performed in a smart mirror, the sense of immersion of a person of interest who is provided with a service may be improved. For another example, when automatic video alignment is performed in an intelligent video surveillance system, convenience of a manager who checks a surveillance video through an integrated control device may be improved.

In addition, according to the present invention described above, a person of interest may be determined in consideration of demographic characteristic information, information on objects in possession, situation information, and the like among persons recognized in the input image. Through this, a person of interest can be determined with high accuracy.

According to the present invention described above, a customized service screen configured for a person of interest may be provided. Accordingly, satisfaction of a user receiving a service may be improved.

The effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram of an image alignment system according to a first embodiment of the present invention.
2A and 2B are exemplary diagrams of image alignment that may be referred to in some embodiments of the present invention.
3 is a block diagram showing a smart mirror 200 that is a component of the image alignment system according to the first embodiment.
4 is a block diagram of an image alignment system according to a second embodiment of the present invention.
5A and 5B are exemplary diagrams of image alignment that may be referred to in some embodiments of the present invention.
7 is a flowchart of an image alignment method according to an embodiment of the present invention.
8 to 14 are diagrams for explaining a method of determining a person of interest according to some embodiments of the present invention.
15 and 16 are diagrams for explaining a method of displaying a result image after image alignment has been performed, according to some embodiments of the present invention.
17 to 19B are diagrams for explaining a method of configuring a user-targeted service providing screen according to the first embodiment of the present invention.
20 is a diagram for explaining a method of configuring a user-targeted service providing screen according to a second embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, and only these embodiments make the disclosure of the present invention complete, and are common knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to those who have, and the invention is only defined by the scope of the claims. The same reference numerals refer to the same elements throughout the specification.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically. The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase.

As used in the specification, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, actions and/or elements in which the recited component, step, operation and/or element is Or does not preclude additions.

Prior to the description of the present specification, some terms used in the present specification will be clarified.

In the present specification, image alignment refers to rotation, shifting, auto focusing, zoom, crop, and out-of-focusing with respect to a first image according to a predetermined criterion. It means outputting a second image by performing image processing such as focusing). For example, the image alignment may mean outputting a second image by performing the image processing based on a person of interest meeting a predetermined criterion among at least one person included in the first image. Hereinafter, for convenience of description, the first image is referred to as "input image", and the second image is referred to as "alignment image" or "result image".

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings.

First, an image alignment system according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 3.

1 is a block diagram of an image alignment system according to a first embodiment of the present invention. Specifically, FIG. 1 shows a smart mirror system as an image alignment system according to the first embodiment.

Referring to FIG. 1, the smart mirror system may be configured to include a smart mirror 200 and a service providing server 300. However, this is only a preferred embodiment for achieving the object of the present invention, and of course, some components may be added or deleted as necessary. In addition, it should be noted that each of the components of the smart mirror system illustrated in FIG. 1 represents functional elements that are functionally divided, and at least one component may be implemented in a form in which they are integrated with each other in an actual physical environment. Hereinafter, each component of the smart mirror system will be described.

In the smart mirror system, the smart mirror 200 is a computing device in which a half mirror having a translucent function and a reflective function is placed in front and a display panel is disposed at the rear thereof. When an image is output from the display panel It is a computing device that transmits an image through a half mirror and displays an image to the outside and provides a mirror function when there is no image output.

According to an embodiment of the present invention, the smart mirror 200 generates an input image by photographing at least one person through a camera sensor attached to the front, determines a person of interest among the at least one person, and Image alignment may be performed on the input image based on. Only in such an embodiment, the smart mirror 200 may be referred to as “image alignment device”. Hereinafter, in order to provide convenience of understanding, it is assumed that there are a plurality of people or objects included in the input image. However, the scope of the present invention is not limited thereto. That is, even if only one first person appears in the input image, whether the first person is a person of interest may be determined according to whether the first person meets a predetermined criterion.

For example, as shown in FIGS. 2A and 2B, the smart mirror 200 determines a person of interest 11 meeting a predetermined criterion among a plurality of people included in the input image 10a, and The alignment image 10b on which the image alignment has been performed based on (11) may be displayed in various ways. 2B shows that image processing such as cropping and zooming is performed based on the person of interest 11, but this is only an example of image alignment, and the image alignment includes various image processing such as rotation, shifting, and out-focusing. It can be done using a technique. According to the present embodiment, since an image focused on the basis of the user determined as the person of interest is provided through the smart mirror 200, the user's sense of immersion can be increased. A detailed description of the present embodiment will be described later with reference to FIGS. 7 to 16.

According to an embodiment of the present invention, the smart mirror 200 may configure and display a service providing screen targeted to a user determined as a person of interest. For example, if the person of interest is a child who brush teeth with a toothbrush, the smart mirror 200 may configure and display a screen that provides guide content for brushing teeth. Here, the brushing guide content may be content previously stored in the smart mirror 200 or content received from the service providing server 300. Only in such an embodiment, the smart mirror 200 may be referred to as a “service providing device”. According to the present embodiment, since a customized service screen is provided to the user, the user's service satisfaction may be improved. A detailed description of this embodiment will be described later with reference to FIGS. 17 to 20.

According to an embodiment of the present invention, the smart mirror 200 may extract various metadata through image analysis of an input image, and determine a person of interest based on the metadata. In this case, the metadata may be various, for example, information on demographic characteristics of a person detected in an input image, information on an object possessed by a person, situation information, and the like. According to an embodiment, the image analysis is performed by the service providing server 300, and the smart mirror 200 may operate in a manner that receives metadata extracted as a result of the image analysis from the service providing server 300. have. A detailed description of the present embodiment will be described later with reference to FIGS. 8 to 14.

In the smart mirror system, the service providing server 300 is a computing device that supports provision of various services provided based on the smart mirror 200. Here, the computing device may be a notebook, a desktop, a laptop, etc., but is not limited thereto, and may include all types of devices equipped with a computing function and a communication function.

The service providing server 300 may perform image analysis on an input image received from the smart mirror 200 or may provide customized content for a person of interest. For example, if the person of interest is a child holding a toothbrush, the service providing server 300 may provide toothbrushing guide content. For another example, when the person of interest is a woman receiving beauty services, the service providing server 300 may provide various hair style contents to the smart mirror 200.

Each component of the smart mirror system shown in FIG. 1 may communicate through a network. Here, the network is a local area network (LAN), a wide area network (WAN), a mobile radio communication network (mobile radio communication network), Wibro (Wireless Broadband Internet) of all types of wired / wireless networks, etc. Can be implemented.

Hereinafter, the configuration and operation of the smart mirror 200, which is a component of the smart mirror system, will be further described.

3 is an exemplary block diagram illustrating the smart mirror 200.

Referring to FIG. 3, the smart mirror 200 may be configured to include a sensor unit 210, a storage unit 230, a communication unit 250, a display unit 270, and a control unit 290. However, only the components related to the embodiment of the present invention are shown in FIG. 3. Accordingly, those of ordinary skill in the art to which the present invention pertains can recognize that other general-purpose components may be further included in addition to the components illustrated in FIG. 3.

Looking at each component, the sensor unit 210 may include a camera sensor attached to the front surface of the smart mirror 200 and photographing a surrounding image and/or a proximity sensor that detects the proximity of an object. For example, the camera sensor may be controlled by the controller 290 to capture an image only when the proximity of a predetermined object is detected by the proximity sensor.

The storage unit 230 may non-temporarily store one or more computer programs that perform various operations of the smart mirror 200. The storage unit 230 is a nonvolatile memory such as a ROM (Read Only Memory), PROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, etc., a hard disk, a removable disk, or in the technical field to which the present invention belongs. It may be configured to include any well-known computer-readable recording medium.

The communication unit 250 performs data communication with other components of the smart mirror system. To this end, the communication unit 250 may include a wired Internet module, a mobile communication module, or a wireless communication module.

The display unit 270 displays an image as a result of performing image alignment based on a person of interest. Alternatively, the display unit 270 displays a service providing screen configured for a person of interest. To this end, the display unit 270 may be configured to include a display module widely known in the art, such as an LCD panel.

In addition, the display unit 270 may be configured to include a mirror including a mirror function that is disposed on the front side and reflects light from the outside to illuminate the shape of an object. The mirror may provide an image display function that transmits an image signal from the display module to be exposed to the outside. For example, the mirror may be a half mirror. When the light source of the display module is turned off, the half mirror acts like a mirror, and when the light source is turned on, the image of the display module is transmitted to be displayed outside.

The display module may have the same size as the mirror, but may have a size smaller than the mirror so that light is transmitted through only a partial area of the mirror. In the latter case, the mirror may be divided into a first mirror area providing only a mirror function and a second mirror area providing a mirror function or an image display function.

The controller 290 controls the overall operation of each component of the smart mirror 200. The control unit may include a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), or any type of processor well known in the art. In addition, the controller 290 may perform an operation on at least one application or program for executing the method according to embodiments of the present invention, which will be described later.

Specifically, the controller 290 may perform an image alignment method according to an embodiment of the present invention to determine a person of interest from an input image, perform image alignment based on the person of interest, and display an alignment image. A detailed description of the image alignment method will be described later with reference to FIGS. 7 to 16.

The controller 290 may perform a method of configuring a service providing screen targeted to a user according to an embodiment of the present invention in order to determine a person of interest from the input image and construct a service providing screen for the interested person. A detailed description of the method of configuring the service providing screen will be described later with reference to FIGS. 17 to 20.

Each component of FIG. 3 may mean software or hardware such as a Field Programmable Gate Array (FPGA) or an Application-Specific Integrated Circuit (ASIC). However, the components are not limited to software or hardware, and may be configured to be in an addressable storage medium, or may be configured to execute one or more processors. The functions provided in the components may be implemented by more subdivided components, or may be implemented as one component that performs a specific function by combining a plurality of components.

So far, a smart mirror system, which is an image alignment system according to a first embodiment of the present invention, has been described with reference to FIGS. 1 to 3. Next, an image alignment system according to a second embodiment of the present invention will be described with reference to FIGS. 4 to 6.

4 is a block diagram of an image alignment system according to a second embodiment of the present invention. Specifically, FIG. 4 shows an intelligent video surveillance system as the video alignment system according to the second embodiment.

Referring to FIG. 4, the intelligent image monitoring system may be configured to include a plurality of image capturing devices 400a to 400n, an intelligent image analysis device 500 and an image alignment device 600. However, this is only a preferred embodiment for achieving the object of the present invention, and of course, some components may be added or deleted as necessary. In addition, it should be noted that each of the components of the intelligent video surveillance system shown in FIG. 4 represents functional elements that are functionally divided, and at least one component may be implemented in a form in which they are integrated with each other in an actual physical environment. Hereinafter, each component of the intelligent video surveillance system will be described.

In the intelligent video surveillance system, the video capture devices 400a to 400n are devices that capture a designated surveillance area and generate a surveillance image. The image capturing devices 400a to 400n transmit the generated surveillance image to the intelligent image analysis device 500. The image capturing apparatuses 400a to 400n may be implemented as, for example, CCTV, but are not limited thereto.

In the intelligent video surveillance system, the intelligent video analysis device 500 performs intelligent video analysis based on the surveillance images provided by the video capture devices 400a to 400n. For example, the intelligent image analysis apparatus 500 may detect an object of interest through analysis of the surveillance image, and perform recognition and tracking of the object of interest.

In addition, the intelligent image analysis apparatus 500 may detect various events set by an administrator through processes such as detection, recognition, and tracking of the object of interest. For example, the intelligent image analysis apparatus 500 may detect various events such as violation of traffic laws, recognition of wanted persons, recognition of lost children, and the like.

In the intelligent video surveillance system, the image alignment device 600 is a computing device that performs image alignment and displays a surveillance image provided by the image capturing devices 400a to 400n as an input image. Here, the computing device may be a notebook, a desktop, a laptop, etc., but is not limited thereto, and may include all types of devices equipped with a computing function and a communication function.

In the intelligent video surveillance system, the video alignment device 600 may be implemented as an integrated control device. In this case, the image alignment device 600 manages a user interface for setting a detection target event, a result analyzed by the intelligent image analysis device 500, and a surveillance image acquired by the image capturing devices 400a to 400n. Can be provided to.

According to an embodiment of the present invention, the image alignment apparatus 600 may determine a person of interest from among at least one person recognized in the surveillance image, and perform image alignment on the surveillance image based on the person of interest.

For example, as shown in FIGS. 5A and 5B, the image alignment device 600 determines a person of interest 21, 23 meeting a predetermined criterion among a plurality of people included in the surveillance image 20a, and , An image 20b on which image alignment has been performed based on the interested person 21 and 23 may be displayed. In particular, FIG. 5A shows an infant 23 and an adult male 21 satisfying the above condition when a condition in which an adult male appears around an infant for a certain time or longer is set as a detection target event in a surveillance area where infant kidnapping accidents frequently occur. An example is shown in which the person of interest is determined. In addition, FIG. 5B shows an example in which an alignment image 20b, in which image alignment has been performed, based on an infant 23 and an adult male 21 determined as a person of interest, is displayed in each divided area together with the surveillance image 20a. I'm doing it. In this way, the image alignment apparatus 600 may determine a person of interest based on a specific situation and automatically generate an alignment image.

A detailed description of a method of performing image alignment by the image alignment apparatus 600 and a method of displaying an alignment image will be described later with reference to FIGS. 7 to 16.

Each component of the intelligent video surveillance system shown in FIG. 4 may communicate through a network. Here, the network is a local area network (LAN), a wide area network (WAN), a mobile radio communication network (mobile radio communication network), Wibro (Wireless Broadband Internet) of all types of wired / wireless networks, etc. Can be implemented.

Hereinafter, the configuration and operation of the image alignment device 600, which is a component of the intelligent video surveillance system, will be described with reference to FIG. 6.

6 is a hardware configuration diagram of the image alignment apparatus 600.

Referring to FIG. 6, the image alignment apparatus 600 loads a computer program executed by one or more processors 610, bus 630, network interface 640, display 650, and processor 610. ) And a storage 660 for storing the image alignment software 661. However, only the components related to the embodiment of the present invention are shown in FIG. 6. Accordingly, those of ordinary skill in the art to which the present invention pertains can recognize that other general-purpose components may be further included in addition to the components shown in FIG. 6.

The processor 610 controls the overall operation of each component of the image alignment device 600. The processor 610 includes a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), or any type of processor well known in the technical field of the present invention. Can be. Also, the processor 610 may perform an operation on at least one application or program for executing the method according to the embodiments of the present invention. The image alignment apparatus 600 may include one or more processors.

The memory 620 stores various types of data, commands, and/or information. The memory 620 may load one or more programs 661 from the storage 660 in order to execute the image alignment method according to embodiments of the present invention. In FIG. 6, a RAM is illustrated as an example of the memory 620.

The bus 630 provides a communication function between components of the image alignment apparatus 600. The bus 630 may be implemented as various types of buses such as an address bus, a data bus, and a control bus.

The network interface 640 supports wired/wireless Internet communication of the image alignment device 600. In addition, the network interface 640 may support various communication methods other than Internet communication. To this end, the network interface 640 may be configured to include a communication module well known in the art.

The storage 660 may non-temporarily store the one or more programs 661. In FIG. 6, an image alignment software 661 is shown as an example of the one or more programs 661.

The storage 660 is a nonvolatile memory such as a ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, etc., a hard disk, a removable disk, or well in the technical field to which the present invention pertains. It may be configured to include any known computer-readable recording medium.

The image alignment software 661 may execute an image alignment method according to an embodiment of the present invention. For example, the image alignment software 661 is loaded into the memory 620 and, by one or more processors 610, an operation for recognizing at least one person in the input image, and determining a person of interest among the recognized people. The image alignment may be performed with respect to the operation and the input image based on the person of interest, but the image alignment may be performed without a user's input for the image alignment.

For reference, the hardware configuration of the apparatus for providing a service according to an embodiment of the present invention may also be implemented in the configuration shown in FIG. 6. In this case, a computer program for executing a method of configuring a service providing screen targeted to a user may be stored in the storage of the service providing device, and the computer program may be loaded into a memory and executed by a processor.

So far, an intelligent video surveillance system, which is an image alignment system according to a second embodiment of the present invention, has been described with reference to FIGS. 4 to 6. Next, an image alignment method according to an embodiment of the present invention will be described in detail with reference to FIGS. 7 to 16.

Each step of the image alignment method according to an embodiment of the present invention to be described below may be performed by a computing device. For example, the computing device may be an image alignment device 200 or 600. However, for convenience of explanation, the description of the operation subject of each step included in the image alignment method may be omitted. In addition, each step of the image alignment method may be implemented as an operation performed by the image alignment apparatuses 200 and 600 by executing image alignment software by a processor.

7 is a flowchart of an image alignment method according to an embodiment of the present invention. However, this is only a preferred embodiment for achieving the object of the present invention, and of course, some steps may be added or deleted as necessary. Hereinafter, in order to provide convenience of understanding, it is assumed that there are a plurality of people included in the input image. However, as described above, the scope of the present invention is not limited thereto.

Referring to FIG. 7, first, a plurality of people are recognized in an input image (S110). In order to recognize the plurality of people, at least one image processing algorithm or image analysis algorithm widely known in the art may be used.

Next, an interested person is determined among the recognized plurality of people (S130). The interested person may be determined based on at least one of, for example, demographic characteristic information of the recognized person, object information possessed by the recognized person, and situation information. A detailed description of this step (S130) will be described in detail later with reference to FIGS. 8 to 14.

Next, image alignment is performed based on a person of interest without a user's input (S150). As described above, the image alignment may be performed using various image processing techniques such as rotation, shifting, cropping, and zoom.

Next, an image as a result of performing image alignment is displayed (S170). The display method may vary according to exemplary embodiments. A detailed description of this step (S170) will be described in detail with reference to FIGS. 15 and 16.

So far, an image alignment method according to an embodiment of the present invention has been described with reference to FIG. 7. According to the above-described method, an interested person may be determined according to a predetermined criterion, and an image may be provided as a result of automatically performing image alignment based on the interested person. Accordingly, the immersion of the user who is provided with the image is increased, and the user's convenience may also be improved since the user's intervention is not required.

Hereinafter, a method of determining a person of interest performed in step S130 will be described in detail with reference to FIGS. 8 to 14.

In one embodiment, a person of interest may be determined based on demographic characteristic information of the recognized person. That is, from among a plurality of people recognized in the input image, a person having a characteristic corresponding to predetermined demographic information may be determined as a person of interest. At this time. The demographic characteristic information may include information such as sex, age, nationality, ethnicity, and the like.

In order to provide convenience of understanding, a specific example of determining a person of interest based on demographic characteristic information will be described with reference to FIG. 8.

Referring to FIG. 8, as a result of analyzing the age of a first person among a plurality of recognized people, the age of the first person is determined as an age that matches a predetermined parental age group (S131), and the recognized As a result of analyzing the age of the second person among a plurality of people, the age of the second person may be determined as an age matching a predetermined child age range (S133). In this case, the second person may be determined as a person of interest (S135). That is, when a parent and a child are included in the input image, the child may be determined as a person of interest.

In an embodiment, as shown in FIGS. 9A and 9B, a person of interest may be determined based on the size of the face area. Specifically, face regions 32, 34, 36 are extracted for each of the plurality of people 31, 33, and 35 recognized in the input image 30a, and the recognized plurality of people 31, 33, 35 The person 33 having the largest size of the extracted face regions 32, 34, 36 may be determined as the person of interest. In addition, an image 30b may be provided as a result of performing image alignment based on the person of interest 33. According to the present embodiment, since a person located close to the smart mirror 200 is determined as a person of interest, the smart mirror 200 may operate so that a predetermined service is provided in order according to the position of the person.

In an embodiment, a person of interest may be determined from among a plurality of recognized persons who have a predetermined object of interest. Here, the designated object of interest may be variously designated according to the purpose of the system, such as a user terminal, a smart toothbrush, and weapons such as a knife or a pistol. Specifically, an object having a shape of a designated object of interest is detected at each designated location among a plurality of recognized people, and an object having a shape of a designated object of interest at a designated location of a first person is detected among the recognized plurality of people. If so, the first person may be determined as the person of interest. In this case, when the designated object of interest is a toothbrush-shaped object, the designated position may be a position corresponding to the tip of a person's hand.

In order to provide convenience of understanding, it will be further described with reference to FIGS. 10A and 10B. When the designated object of interest is a toothbrush or a smart toothbrush, it is determined whether a toothbrush-shaped object is detected at the fingertips of the plurality of people 41 and 43 recognized in the input image 40a. As a result of the determination, when the toothbrush-shaped object 44 is detected at the fingertip of the first person 43, as shown in FIG. 10B, the result image of performing image alignment based on the first person 43 ( 40b) is provided.

In the above-described embodiment, there may be a case where there are a plurality of people who have a designated object of interest. In this case, at least one person may be determined as the person of interest according to various criteria.

For example, at least one person located at a close distance may be determined as the person of interest based on the size of the face area.

For another example, the person of interest may be determined based on the order in which the input image is entered. Specifically, when an object having the shape of the designated object of interest is detected at a designated location of the first person, and an object having the shape of the designated object of interest is detected at the designated location of the second person, the first person and the first person Among 2 people, a person who first enters the input image may be determined as the person of interest. Of course, a plurality of people may be determined as interested people according to the order of entry.

For another example, a person located in a predetermined direction among a plurality of people holding a designated object of interest may be determined as the person of interest. Specifically, when an object having the shape of the designated object of interest is detected at a designated location of the first person, and an object having the shape of the designated object of interest is detected at the designated location of the second person, the first person and the first person Among the 2 people, a person located in a predetermined direction may be determined as the person of interest. According to this example, when a smart mirror is installed in the wash basin of a kindergarten, and a plurality of kindergarten students enter from the right, brush their teeth according to the brushing guide content, and exit to the left, a brushing guide service based on image alignment in order according to the entrance order. Can be provided.

In an exemplary embodiment, a person of interest, among a plurality of persons holding a designated object of interest, may be determined as a person of interest. Here, the designated position may vary according to the type of the object of interest. In order to provide convenience of understanding, a case where the object of interest is a smart toothbrush will be further described with reference to FIG. 11, for example.

Referring to FIG. 11, among a plurality of children 51, 53, 55 detected by the smart toothbrush 52, 54, 56 at the fingertips, the child 53 detected by the smart toothbrush around the mouth will be determined as a person of interest. I can. Through this, the smart mirror 200 can be operated so that the image and tooth brushing guide contents are provided as a result of automatically performing image alignment to the child who first started brushing among children who are provided with a brushing guide service through the smart mirror 200. have.

In some embodiments, as shown in FIG. 12, voice recognition is performed on a voice included in an input image or a voice signal 61 sensed through a separate voice sensor, and the text 63 derived through the voice recognition is performed. ) Can be used to determine the person of interest. Hereinafter, some embodiments based on speech recognition will be described.

In an embodiment, when a text corresponding to the recognized voice is detected in an area inside the outline of the first person, the first person may be determined as a person of interest. With reference to FIG. 13, a case where a teacher calls a first child among a plurality of children with a name tag will be described further by way of example. Referring to FIG. 13, when text indicating the name of the first child is obtained by recognizing a voice signal calling for a first child, the inner areas 71 and 73 of the plurality of people included in the input image 70 At 75), it may be determined whether the acquired text is detected. When the text 74 identical to the acquired text is recognized in the area 73 inside the outline of the first child among the plurality of people, the first child may be determined as a person of interest. Accordingly, a result of image alignment based on a child wearing a name tag with a called name among a plurality of children may be provided.

In one embodiment, when the text corresponding to the recognized voice is the name of the first person, face recognition is performed on each of the plurality of recognized persons, and the recognized plurality of persons among the plurality of recognized persons based on the face recognition result The first person can be determined as the person of interest. Specifically, when the first person's name is acquired as text through speech recognition, face recognition is performed for each of a plurality of people recognized using a pre-built face DB including the person's name information and a face image. . Then, a person whose face corresponding to the name of the first person is recognized among the plurality of recognized persons is determined as the person of interest. According to the present embodiment, image alignment is performed based on a specific person whose name is called, and this can be utilized in various cases. For example, if the parents of a missing child in the surveillance area search for the child by calling the child's name, it is used for a missing child search service that can automatically align the video for the video that includes the child in each surveillance video. Can be. For another example, when the police trace the wanted person by calling the name of the wanted person within the surveillance area, video alignment can be automatically performed on the video including the wanted person among each surveillance video, so it may be used for the wanted person arrest service. .

In some embodiments, a person of interest corresponding to a current situation may be determined through a context-based person of interest determination logic for each of a plurality of recognized persons. Here, the context information used in the context-based interest determination logic may include a combination of weather, time, place, and demographic characteristics.

In an embodiment, a situation may be determined based on a combination of demographic characteristics of each of a plurality of recognized people, and a person meeting conditions according to the determined situation may be determined as a person of interest. For example, as shown in FIG. 14, among a plurality of recognized people 81, 83, and 85, a person 83, 85 composed of a combination of a man in his 40s and a woman in his 30s may be determined as a person of interest. Alternatively, a person who meets conditions according to various combinations of parents and children, men and men, and men and women may be determined as the interested person.

Until now, a method of determining a person of interest that can be referred to in some embodiments of the present invention has been described with reference to FIGS. 8 to 14. Hereinafter, a method of displaying an image as a result of performing image alignment based on a person of interest in step S170 will be described.

In an embodiment, an image in which a highlight process has been performed for a person of interest or a person who satisfies a specified condition among the persons of interest may be displayed. Here, the highlight processing may include all image processing techniques that are processed so that a person of interest stands out, such as a bounding box processing and a color processing for a person of interest while the surrounding is black and white. For a more specific example, when the brushing guide service is provided through the smart mirror 200, the first child who is performing brushing teeth is processed in color, and the child who has not started brushing teeth is processed in black and white, and the resulting image will be provided. I can.

In one embodiment, predetermined additional information about a person of interest may be further displayed. For example, when an action meeting a specified condition is detected from a person of interest, information about a period in which the action is performed may be displayed around the person of interest. For a more specific example, information such as a time when a child brush his teeth, and a time when a person suspected of a kidnapping lingers around an infant may be displayed around the person. In addition, various information such as demographic characteristic information of the interested person may be further displayed around the interested person.

In one embodiment, when a plurality of persons of interest are determined, a first result of image alignment performed based on a first interested person is displayed in the first area, and a second result of image alignment performed based on a second interested person An image may be displayed in the second area. For example, as shown in FIG. 15, when a plurality of children 91, 93, and 95 brushing teeth are determined to be interested persons, image alignment is performed based on each child (91, 93, 95) Then, each result image may be displayed in the divided regions 90a, 90b, and 90c.

In the above-described embodiment, the size of the divided area may be dynamically adjusted according to a predetermined score or priority. For example, as shown in FIG. 16, when a plurality of children 101, 103, and 105 brush their teeth, the size of the divided regions 100a, 100b, and 100c based on the brushing score of each child is Can be adjusted. In particular, FIG. 16 shows as an example a case in which the second child 63 is the highest in the order of tooth brushing scores, followed by the first child 61 and the third child 65. Here, the brushing score is, for example, information received from a smart toothbrush used by each child (101, 103, 105), image analysis information indicating whether or not to brush teeth according to the brushing guide content when the guide content is provided ( eg Calculated based on the degree of correspondence between the content image and the captured image), it may be calculated in various ways based on the brushing time. According to this example, the effect of inducing correct brushing of teeth to children may be generated by stimulating the competitive psychology of children.

So far, an image alignment method according to an embodiment of the present invention has been described with reference to FIGS. 7 to 16. Next, a method of configuring a user-targeted service providing screen according to some embodiments of the present invention will be described with reference to FIGS. 17 to 20.

In the method of configuring the service providing screen to be described later, each step may be performed by a computing device. For example, the computing device may be a service providing device. However, for convenience of explanation, the description of the operation subject of each step included in the method of configuring the service providing screen may be omitted. In addition, each step of the method for configuring the service providing screen may be an operation performed in the service providing device by executing the configuration software of the service providing screen targeted to the user by the processor.

First, a method of configuring a user-targeted service providing screen according to a first embodiment of the present invention will be described with reference to FIGS. 17 to 19B.

17 is a flowchart illustrating a method of configuring a service providing screen targeted to a user according to the first embodiment. However, this is only a preferred embodiment for achieving the object of the present invention, and of course, some steps may be added or deleted as necessary. Hereinafter, in order to provide convenience of understanding, it is assumed that there are a plurality of people included in the input image. However, as described above, the scope of the present invention is not limited thereto.

Referring to FIG. 17, a plurality of people are recognized in an input image (S210), and a person of interest is determined from among the recognized plurality of people (S230). Steps (S210, S230) are the same as the above-described steps (S110, S130) to be omitted in order to exclude redundant description.

Next, a screen for providing a service targeting a person of interest is constructed and displayed without a user input (S250). For example, if the person of interest is a child with a smart toothbrush, toothbrushing guide content may be displayed to provide a toothbrushing guide service.

In an embodiment, a service provision screen may be configured as shown in FIG. 18A. Specifically, when there are a plurality of interested persons 111, 113, 115, an image is displayed as a result of image alignment based on the interested person in each divided area 110a, 110b, 110c, and content for each interested person A service providing screen may be configured such that is displayed on some areas 112, 114, and 116 in each of the divided areas 110a, 110b, and 110c. Depending on the embodiment, the resulting image is not displayed, and each content may be displayed in each of the divided regions 110a, 110b, and 110c.

In an embodiment, as illustrated in FIG. 18B, when common content is provided to a plurality of interested persons 111, 113, and 115, the common content may be displayed in a new divided area 110d. In FIG. 18B, the new divided area 110d is shown as an example located at the bottom, but the size and position of the new divided area 110d may be changed as much as possible according to the resolution of the content.

Meanwhile, according to an embodiment of the present invention, a service providing screen may be configured using data received from a user terminal of a person of interest. In more detail, when the user terminal or the person whose object having the shape of the specified object is detected is determined as the person of interest, the identification information (eg ID, name, etc.) of the person of interest is provided through a previously established face DB or identification information DB. Can be obtained. Then, information on the user terminal held by the interested person is inquired using the identification information, and a service provision screen can be constructed by receiving predetermined content from the user terminal using the information on the user terminal. have. In this case, the information on the user terminal corresponding to the identification information may be pre-stored information. In order to provide a more convenient understanding, the present embodiment will be further described with reference to FIGS. 19A and 19B.

19A and 19B illustrate an example in which a speaker is determined as a person of interest, and presentation content received from a terminal of the speaker is displayed through a service providing apparatus. Hereinafter, it will be described with reference to FIGS. 19A and 19B.

Assuming that the service providing device is a computing device installed at the presentation place, the service providing device is interested in the person 121 whose pointer-shaped object 122 is detected at the tip of a hand among a plurality of people recognized in the input image 120a. It can be decided by a person.

When a person of interest is determined, as shown in FIG. 19B, the service providing device generates an image 120b as a result of performing image alignment based on the person of interest 121 and displays it in a first area on the screen. The presentation content 123 received from the user terminal 130 possessed by the person 121 may be displayed in the second area on the screen. 19B illustrates an example in which the result image 120b and the content 123 are displayed in each independently divided area, but as described above, the layout of the service screen, the display method, and the like may vary.

According to the example described with reference to FIGS. 19A and 19B, the presenter is automatically determined as a person of interest, and a customized service screen may be automatically configured so that the presenter can proceed with the presentation. Accordingly, user convenience can be greatly improved.

So far, a method of configuring a user-targeted service providing screen according to the first embodiment of the present invention has been described with reference to FIGS. 17 to 19B. Hereinafter, a method of configuring a user-targeted service providing screen according to a second embodiment of the present invention will be described with reference to FIG. 20.

20 is a flowchart illustrating a method of configuring a user-targeted service providing screen according to the second embodiment. However, this is only a preferred embodiment for achieving the object of the present invention, and of course, some steps may be added or deleted as necessary.

Referring to FIG. 20, a plurality of people are recognized in an input image (S310).

Next, biometric data is received from the recognized biometric signal sensors of a plurality of people (S330). Here, the biosignal sensor may include, for example, a biosignal sensor attached to a user's body, a wearable device, and/or a biosignal sensor attached to a user terminal. In addition, the biometric data may include heart rate, blood pressure, oxygen saturation, stress index, brushing degree, and the like.

Next, a person of interest is determined from among the plurality of recognized persons using the received biometric data (S350). For example, a person whose biometric data is not in the normal category, such as a person with high blood pressure, a person who does not brush teeth well, and a person with a high heart rate, may be determined as a person of interest.

In an embodiment, an image analysis result of a plurality of recognized people may be further used to determine the person of interest. For example, in the case of determining a person with an abnormality in health as a person of interest, analysis information such as blood flow information on the face and facial expression information (eg frowned expression) is derived through image analysis, and the derived information is further used. More precisely, the person of interest can be determined.

Next, a screen for providing a service targeting a person of interest is constructed and displayed without a user input (S370). For example, when a person having an abnormality in health is determined as a person of interest, exercise content, content for a specific disease, and the like may be provided. Alternatively, when a person who does not brush teeth well is determined as a person of interest, guide content for brushing teeth may be provided.

So far, a method of configuring a user-targeted service providing screen according to a second embodiment of the present invention has been described with reference to FIG. 20. As described above, a customized service may be provided by using the user's biosignal information.

The concept of the present invention described with reference to FIGS. 1 to 20 so far may be implemented as computer-readable codes on a computer-readable medium. The computer-readable recording medium is, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). I can. The computer program recorded on the computer-readable recording medium may be transmitted to another computing device through a network such as the Internet and installed in the other computing device, thereby being used in the other computing device.

Although the operations are illustrated in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order shown or in a sequential order, or all illustrated operations must be executed to obtain a desired result. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the above-described embodiments should not be understood as necessitating such separation, and the program components and systems described are generally integrated together into a single software product or may be packaged into multiple software products. It should be understood that there is.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. I can understand. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting.

Claims (21)

Recognizing, by the computing device, at least one person from the input image;
Determining, by the computing device, a first person of interest and a second person of interest among the recognized persons;
The computing device displays an image as a result of performing image alignment with respect to the input image based on the first person of interest, and displays an image as a result of performing image alignment based on the second person of interest. Displaying in two areas;
Calculating, by the computing device, a score based on actions of the first person of interest and the second person of interest appearing in the input image; And
Adjusting, by the computing device, a size of at least one of the first region and the second region based on scores calculated for each of the first person of interest and the second person of interest.
Containing,
Video alignment method. The method of claim 1,
Determining the first person of interest and the second person of interest,
Determining a person of interest based on a demographic characteristic of each of the recognized persons, an object possessed by each of the recognized persons, or a result of applying a context-based person of interest determination logic to each of the recognized persons. ,
Video alignment method. delete The method of claim 1,
The step of calculating a score based on the behavior of each of the first interested person and the second interested person,
Calculating a degree to which the image of the first person of interest matches a predetermined reference image; And
Calculating a degree to which the image of the second person of interest matches the predetermined reference image
Containing,
Video alignment method. The method of claim 1,
Adjusting the size of at least one of the first area and the second area,
Enlarging the size of the first area based on the determination that the score calculated for the first interested person is higher than the score calculated for the second interested person,
Video alignment method. The method of claim 1,
Adjusting the size of at least one of the first area and the second area,
Reducing the size of the second area based on the determination that the score calculated for the first interested person is higher than the score calculated for the second interested person,
Video alignment method. The method of claim 1,
Displaying a score calculated for each of the first interested person and the second interested person
Further comprising,
Video alignment method. Combined with a computing device,
Recognizing at least one person from the input image;
Determining a first person of interest and a second person of interest among the recognized persons;
Displaying an image as a result of performing image alignment with respect to the input image based on the first person of interest, and displaying an image as a result of performing image alignment based on the second person of interest. step;
Calculating a score based on the actions of each of the first person of interest and the second person of interest appearing in the input image; And
Adjusting the size of at least one of the first region and the second region based on the scores calculated for each of the first person of interest and the second person of interest
A computer program stored in a computer-readable recording medium to execute. A camera sensor that generates an input image;
Recognizing at least one person from the input image, determining a first person of interest and a second person of interest among the recognized people, performing image alignment with respect to the input image based on the first person of interest, and the A processor that performs image alignment based on a second person of interest; And
A display configured to display an image as a result of performing image alignment based on the first person of interest in a first area, and displaying an image as a result of performing image alignment based on the second person of interest in a second area,
The processor,
A score is calculated based on the actions of each of the first person of interest and the second person of interest appearing in the input image, and based on the scores calculated for each of the first person of interest and the second person of interest, the second Adjusting the size of at least one of the first region and the second region,
Computing device. delete delete delete delete delete delete delete delete delete delete delete delete

Images