JP2016036081A - Image processing device, method and program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, a method and program that can combine associated information of a subject so that the information is easily viewed.SOLUTION: A face detector 140 and a face recognizer 142 specify a subject (person) contained in an image. CPU 148 refers to a superposed image management table for storing character data to be displayed at a front side and image data to be displayed at a depth side on a subject basis, and searches for data of a subject which is registered in the superposed image management table and located in the front row out of subjects contained in an input image. A layer generator 146 generates a front-row layer in front of the subject located in the front row, and also generates a deepest layer behind a subject located at the deepest position. CPU 148 controls an OSD unit 128 so that character data of a subject which is registered in the superposed image management table and located in the front row is disposed in the front-row layer, and image data are disposed in the deepest layer.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, method, and program, and more specifically, to an image processing apparatus, method, program, and recording medium that superimpose information about a subject included in an image on a moving image.

  In recent years, cameras capable of displaying a photographed subject in three dimensions have been commercialized. Stereoscopic display is realized by acquiring depth information of the subject and adding appropriate parallax to the right-eye image and the left-eye image. In addition, a technique for generating a layer at a desired depth of a stereoscopic video and superimposing an animation is also known.

  Patent Document 1 describes a system that acquires the depth of a target position from a recorded stereoscopic image and superimposes a telop or the like based on the acquired depth.

JP 2011-010128 A

  In the technique described in Patent Document 1, when a plurality of positions of interest are set, a plurality of telop displays are performed, and there is a problem that the display becomes difficult to see. In addition, if the change in the depth direction frequently occurs in a situation where a plurality of positions of interest are set, the telop display becomes difficult to see.

  There is a demand to display information on the subject effectively on the near side and the far side of the subject of interest, but the technique described in Patent Literature 1 cannot meet this demand.

  It is an object of the present invention to present an image processing apparatus, method, program, and recording medium that meet such demands.

  An image processing apparatus according to the present invention includes a subject detection unit that detects a subject included in an input image, a depth detection unit that detects the depth of the subject, and related data to be displayed on the near side and a back side for each subject. The related data management table for storing the related data to be displayed on the screen and the related data management table, and among the subjects included in the input image, the subject registered in the related data management table is closest to the subject. A search unit that searches for related data of a subject that is positioned; and a frontmost layer of the input image that is closest to the subject that is closest to the front, and a deepest layer that is deeper than the subject that is positioned deepest Layer generation means and related data to be displayed on the near side of the subject searched by the search means are arranged on the foremost layer and should be displayed on the far side. And having a means for placing the associated data to the innermost layer.

  According to the present invention, data related to the foreground subject is arranged in the foreground layer and the backmost layer, respectively, so that data corresponding to the subject can be added even when the subject is switched in the depth direction.

It is a schematic block diagram of one Example of this invention. It is explanatory drawing of the layer arrangement | positioning in a present Example. It is an example of the superimposition image management table in a present Example. It is an operation | movement flowchart of the registration procedure to a superimposition image management table. It is explanatory drawing of the distance relationship from a camera to a to-be-photographed object. It is an operation | movement flowchart at the time of imaging | photography in a present Example. It is an example of a display in a present Example. It is another operation | movement flowchart at the time of imaging | photography.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows a schematic block diagram of an imaging apparatus equipped with an embodiment of an image processing apparatus according to the present invention.

  The lens unit 102 of the image pickup apparatus 100 has a function of correcting an imaging position moved by movement of a fixed lens group for focusing, a variable power lens group, a diaphragm, and a variable power lens group, and a function of performing focus adjustment. It is composed of a correction lens group having both. An object image is formed on the imaging surface of the image sensor 104 by the lens unit 102. The image sensor 104 is composed of, for example, a CCD (Charge Coupled Device) image sensor, and generates an image signal representing a subject image by converting light intensity into electric charge. The A / D processing unit 106 performs predetermined processing on the output image signal of the image sensor 104 and outputs a digital image signal. The lens unit 102, the image sensor 104, and the A / D processing unit 106 are collectively referred to as a camera unit 108. Although not shown in FIG. 1, the camera unit 108 includes an actuator for driving a variable power lens group and a diaphragm, a sensor for correcting camera shake (for example, an angular velocity sensor) and its correcting means (such as a shift lens). ).

  A camera control CPU (Central Processing Unit) 110 controls the camera unit 108 according to the control of the CPU 148. In addition, the camera control CPU 110 transmits information obtained from the camera unit, such as focusing information and camera shake information, to the CPU 148.

  The microphone unit 112 is an audio input unit that captures ambient audio during shooting, and includes an amplifier that amplifies the input audio and a band limiting filter that limits the bandwidth. The A / D processing unit 114 converts the output audio signal of the microphone unit 112 into a digital audio signal.

  The encoder 116 compresses and encodes the image data from the camera unit 108 and the audio data from the A / D processing unit 114 under the control of the CPU 148 to generate compressed date data in a predetermined format. As a compression method, MPEG (Moving Picture Experts Group) 2 method and other moving image compression methods can be used. The encoder 116 has a function of notifying the CPU 148 of information necessary for conversion between the data position and the frame position.

  The recording / reproducing circuit 118 records the compressed video data from the encoder 116 on the memory card 122 using the memory 120, reads the compressed video data recorded on the memory card 122, and supplies it to the decoder 124. The CPU 148 controls the recording / reproducing operation of the recording / reproducing circuit 118. The memory card 122 is a recording medium for recording captured moving images (including sound), and is connected to the recording / reproducing circuit 118 via a predetermined interface. The recording / reproducing circuit 118 designates the start address and data amount of the memory 120 and the start address of the data write destination of the memory card 122, and transfers data from the memory 120 to the memory card 122 by a DMA (Direct Memory Access) function.

  The decoder 124 decodes the compressed moving image data from the recording / reproducing circuit 118. The decoder 124 supplies audio data and video data obtained by decoding to an audio output unit 126 and an OSD (On Screen Display) circuit 128, respectively. The video data is, for example, ITU-R BT. 656 (CCIR656) or the like. As will be described in detail later, the OSD circuit 128 captures the video data from the decoder 124, reduces it to a desired size, and superimposes information such as a menu screen, title, and time for performing various settings at a specified position. To do. The OSD circuit 128 supplies the video data as a superimposition result to the video output unit 130. For example, a speaker outputs to the audio output unit 126, and a video display device is connected to the video output unit 130.

  An electronic view finder (EVF) 132 is used by a photographer to confirm a subject in the field of view of the camera unit 108, and displays an image output from the camera unit 108 at a constant cycle.

  The liquid crystal panel 134 has a display screen larger than the EVF 132 and is used not only as a viewfinder but also for displaying an output image of the OSD circuit 128. A touch panel 136 is disposed on the display screen of the liquid crystal panel 134. The liquid crystal panel 134 displays operated elements such as thumbnail images and virtual buttons, and the user can select or operate them using the touch panel 136. The screen control unit 138 determines whether or not there is a user operation on the touch panel 136 and whether or not the operation is performed on the operated element, and supplies information indicating the operation content including the operation position coordinates to the CPU 148. Note that the screen control unit 138 can also control virtual buttons displayed on the liquid crystal panel 134 in accordance with the operation content on the touch panel 136.

  The face detection unit 140 functions as a subject detection unit, and performs face detection processing on image data captured by the camera unit 108 (image sensor 104) to detect a human face area included in the image. This face detection process is performed by a known algorithm. For example, the face detection unit 140 extracts feature points such as eye, nose and mouth end points and face contour points from the image data by a process of extracting known feature points, and based on these feature points. Then, the face area and the face size of the subject are detected.

  The face recognition unit 142 generates face authentication data indicating the features of the face to be authenticated based on the detection result of the face detection unit 140. For example, the face recognition unit 142 generates face authentication data from the position of the detected feature point of the face, the size of the face part obtained from the feature point, the relative distance of each feature point, and the like.

  The depth detection unit 144 detects the distance from the imaging device 100 to the subject. For example, the distance from the imaging device 100 can be calculated for each subject and a plurality of points of each subject using two images with different parallaxes obtained by a stereo camera. Also in this embodiment, such a known detection method is adopted.

  The layer generation unit 146 generates a layer in which characters and icons can be arranged at a position corresponding to a predetermined depth. The layer generation method will be described later.

  The CPU 148 controls the entire imaging apparatus 100. The CPU 148 includes a nonvolatile memory (ROM) for storing a program, a volatile memory (RAM) serving as a work area, a timer for measuring time, and a port or interface for transferring data to and from the outside. The EEPROM 150 stores a control program that operates on the CPU 148, various programs including a program that handles the file system of the memory card 122, and constants necessary for the operation.

  The switch operation unit 152 includes various switches and buttons, and supplies a signal corresponding to a user operation to the CPU 148. The switch operation unit 152 and the touch panel 136 allow the user to instruct the CPU 148 to perform a desired operation.

  Each of the above blocks including the CPU 148 can exchange data via the bus 154.

  With reference to FIG. 2, the layer produced | generated by the layer production | generation part 146 is demonstrated. FIG. 2A shows an example of an image within the photographing field of view of the camera unit 108 (display image of the liquid crystal panel 134), and FIG. 2B shows a plan view showing the distance from the imaging device 100. FIG.

  In the example shown in FIG. 2A, a subject (person) 204 is positioned on the near side of the background 202, and a telop 206 indicating the name of the subject 204 is superimposed and displayed in front of the subject 204 by processing described later. The As the background 202, a star or a starry sky is illustrated.

  In the example shown in FIG. 2A, the layer generation unit 146 moves the background layer 212, the subject layer 214, and the telop layer 216 to a position close to the imaging device 100 from a distance as shown in FIG. It generates so that it may be located in order. The background 202 is located on the background layer 212, the subject 204 is located on the subject layer 214, and the telop 206 is located on the telop layer 216. The telop 206 can be animated to scroll on the telop layer 216.

  FIG. 3 shows an example of a superimposed image management table for managing images to be superimposed. The CPU 148 holds the superimposed image management table as a related data management table for storing related data to be displayed on the near side and related data to be displayed on the back side for each subject. The superimposed image management table can generate a layer (front layer) located on the front side (near side) and a rear layer located on the rear side (far side) for each subject. In the example shown in FIG. 3, text data (for example, the name of a subject) is set in the previous layer, and image data is set in the rear layer. A telop layer 216 shown in FIG. 2B is a front layer for the subject 204, and a background layer 212 is a rear layer for the subject 204.

  FIG. 4 shows a flowchart of an operation example for creating the superimposed image management table. With reference to the example shown in FIG. 5, the procedure for creating the superimposed image management table in this embodiment will be described. As an illustrative example, a situation is assumed in which five subjects (persons) are located in the front direction of the imaging apparatus 100 as illustrated in FIG.

  In step S <b> 401, the CPU 148 causes the face detection unit 140 to detect a human face from the captured image and causes the face recognition unit 142 to recognize the detected face. In the example shown in FIG. 5, for example, the face of the foremost subject A is detected and recognized. When someone's face is detected and recognized (S401), the CPU 148 proceeds to S402, and when no person's face is detected (S401), the flow shown in FIG. 5 ends.

  In S402, the CPU 148 checks whether or not the detected and recognized subject has been registered in the superimposed image management table. If registered (S402), the CPU 148 returns to S401, causes the face detection unit 140 to detect the face of the next subject, and causes the face recognition unit 142 to recognize the detected face. At this stage, for example, the face of the subject B is detected and recognized.

  If the detected / recognized subject is not registered in the superimposed image management table (S402), the CPU 148 adds the subject entry to the superimposed image management table in S403, and superimposes the subject on the previous layer and the subsequent layer. Set the text and image to be set. In this embodiment, the CPU 148 allows the user to input the name of the corresponding subject, sets the input character string to the previous layer, and enables animation such as scrolling. The CPU 148 causes the user to arbitrarily select and register the current layer from the current captured image of the same subject and the previously recorded image.

  In the example shown in FIG. 5A, the above processing is repeated for the subjects B, C, and D. Thereby, a superimposed image management table for the subjects A, B, C, and D is prepared. Needless to say, a superimposed image management table created on a computer may be transferred to the imaging apparatus 100.

  FIG. 6 shows a flowchart of data superimposition processing in the shooting mode of the present embodiment. With reference to FIG. 6, the layer processing at the time of recording of the present embodiment will be described based on the example shown in FIG.

  In step S601, the CPU 148 causes the face detection unit 140 to detect a human face from the captured image to be recorded, and causes the face recognition unit 142 to recognize the detected face. In the example shown in FIG. 5A, subjects A, B, C, and D are detected and recognized. When someone's face is detected and recognized (S601), the CPU 148 proceeds to S602, and when no person's face is detected (S601), the flow shown in FIG. 6 ends.

  In step S <b> 602, the CPU 148 uses the depth detection unit 144 to detect the subject located closest to the imaging device 100 and the subject located farthest away. In the example shown in FIG. 5A, the subject A is positioned closest to the subject, and the subject D is positioned farthest.

  In step S <b> 603, the CPU 148 determines whether or not the closest subject (here, subject A) has been registered in the superimposed image management table. For example, the subject A is searched in the superimposed image management table. When the subject located at the forefront is not registered in the superimposed image management table (S603), the CPU 148 returns to S601. When the closest subject has been registered in the superimposed image management table (S603), the CPU 148 selects the data of the previous layer and the rear layer of the subject in S604. In the example shown in FIG. 5A, the CPU 148 selects the text A and the image A by referring to the record related to the subject A in the superimposed image management table.

  In S605, the CPU 148 sets the data of the front layer and the rear layer selected in S604 as the foremost layer and the backmost layer of the captured image, respectively, and returns to S601. The foremost layer is the front layer of the subject located closest to the foreground, and the deepest layer is the rear layer of the subject located farthest. The CPU 148 causes the OSD circuit 128 to execute data arrangement and synthesis on each layer.

  When the subject is switched in the depth direction, the CPU 148 detects the foremost subject in S602, and sets the text and image set for the subject in the foreground layer and the backmost layer, respectively. In the example shown in FIG. 5A, since the subject detected as being closest to the subject is the subject A, the text A is set in the previous layer with respect to the subject A, and the image is displayed in the rear layer of the subject D located farthest away. A is set.

  As described above, information related to the subject located at the foreground is superimposed on the front layer of the subject located at the forefront and the rear layer of the subject located at the farthest. As a result, the user can superimpose and record desired information on the image output from the camera unit 108, and can output the information from the video output unit 130 to the outside. The recorded moving image may be read out from the memory card 122, subjected to the same processing, recorded in the memory card 122, or output to the outside from the video output unit 130.

  FIG. 7 shows a display example of the result of superimposing the related information (telop / image) according to this embodiment. FIG. 7A shows a display example when the subject A is in the foreground, and FIG. 7B shows a table in which the subject C has moved to the forefront from the state shown in FIG. An example is shown.

  In the display example shown in FIG. 7A, information on the previous layer (telop A) 701 related to the subject A is displayed on the near side of the subject A, and information on the rear layer (image A) 702 related to the subject A is displayed. It is superimposed on the rear side of the subject D located farthest away. Here, the image A is a captured face image of the subject A registered in the superimposed image management table.

  In the display example shown in FIG. 7B, the subject C is positioned closest to the front. Therefore, information on the previous layer related to the subject C (telop C) 703 is displayed on the front side of the subject C, and information on the rear layer related to the subject C (image C) 704 is stored in the farthest subject D. It is superimposed on the rear side. Here, the image C is a captured face image of the subject C registered in the superimposed image management table.

  FIG. 8 shows a flowchart of the process of superimposing data on a layer in the shooting mode when an unregistered subject is detected in the foreground in the superimposed image table. As shown in FIG. 5B, this is a case where the unregistered subject E is located at the closest distance to the imaging device 100 in the imaging field of the imaging device 100. In FIG. 5B, subjects A, C, and D are located farther than the subject E.

  In step S <b> 801, the CPU 148 causes the face detection unit 140 to detect a human face from the captured image to be recorded, and causes the face recognition unit 142 to recognize the detected face. In the example shown in FIG. 5B, the subjects E, A, C, and D are detected and recognized in the order closer to the imaging device 100. When someone's face is detected and recognized (S801), the CPU 148 proceeds to S802, and when no person's face is detected (S801), the flow shown in FIG. 8 ends.

  In step S <b> 802, the CPU 148 uses the depth detection unit 144 to detect a subject located closest to the imaging device 100 and a subject located farthest from the imaging apparatus 100. In the example shown in FIG. 5B, the subject E is located closest to the subject, and the subject D is located farthest away.

  In step S803, the CPU 148 determines whether or not the closest subject (here, subject E) has been registered in the superimposed image management table. If the closest object is not registered in the superimposed image management table (S803), the CPU 148 proceeds to S806. When the closest subject has been registered in the superimposed image management table (S803), the CPU 148 selects data of the front layer and the rear layer of the subject in S804. In S805, the CPU 148 sets the data of the front layer and the rear layer selected in S804 as the foremost layer and the backmost layer of the captured image, respectively, and returns to S801. The foremost layer is the front layer of the subject located closest to the foreground, and the deepest layer is the rear layer of the subject located farthest.

  In step S806, the CPU 148 detects the second closest subject (here, the subject A). If the second closest subject cannot be detected (S806), the CPU 148 returns to S801, and if it can be detected (S806), the process proceeds to S807.

  In step S807, the CPU 148 determines whether the subject detected in step S806 has been registered in the superimposed image management table. If not registered in the superimposed image management table (S807), the CPU 148 returns to S801. If registered in the superimposed image management table (S807), the CPU 148 selects the data of the front layer and the rear layer of the second subject in S808. In the example shown in FIG. 5B, text A and image A are selected. In step S805, the CPU 148 sets the data of the front layer and the rear layer selected in step S708 as the foremost layer and the backmost layer of the captured image, and returns to step S801.

  As described above, the text data related to the registered subject detected at the forefront is superimposed on the foremost side and the image data is superimposed on the farthest side. Thereby, even when the information about the predetermined subject of the input image is not registered, the information of different registered subjects can be displayed in a superimposed manner.

  If the subject located second closest is not registered, a registered subject located farther away may be searched for and related information superimposed on it.

  In the embodiment described above, the control by the CPU 148 can be realized by loading a corresponding control program or image processing program into the CPU 148 and executing it. Such a program can be loaded into the imaging apparatus 100 and the computer via a computer-readable recording medium or via a network.

  Although preferred embodiments of the present invention have been described, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof.

Claims (5)

Subject detection means for detecting a subject included in the input image;
Depth detection means for detecting the depth of the subject;
For each subject, a related data management table storing related data to be displayed on the front side and related data to be displayed on the back side;
Search means for referring to the related data management table and searching for related data of a subject that is registered in the related data management table and is closest to the subject among the subjects included in the input image;
A layer generating means for generating a foremost layer in front of a subject positioned closest to the input image and generating a deepest layer deeper than a subject positioned deepest;
The related data to be displayed on the front side of the subject searched by the search means is arranged on the frontmost layer, and the related data to be displayed on the back side is arranged on the innermost layer. An image processing apparatus.   The related data to be displayed on the front side is text data describing a related subject, and the related data to be displayed on the back side is image data indicating a related subject. Image processing apparatus. A subject detection step for detecting a subject included in the input image;
A depth detection step for detecting the depth of the subject;
For each subject, a related data management table storing related data to be displayed on the near side and related data to be displayed on the back side is referred to, and the subject included in the input image is registered in the related data management table. A search step for searching related data of the subject that is the closest to the subject,
A layer generation step of generating a foreground layer in front of a subject positioned closest to the input image, and generating a backmost layer behind a subject positioned deepest in the input image;
Placing related data to be displayed on the near side of the subject searched in the searching step on the foremost layer, and placing related data to be displayed on the back side on the deepest layer. An image processing method characterized by the above.   An image processing program for causing a computer to function as each unit of the image processing apparatus according to claim 1.   A computer-readable recording medium storing the image processing program according to claim 4.

Images