JP2019029942A - Imaging support apparatus, imaging apparatus, communication terminal and imaging support control method - Google Patents

Abstract

To enable a photographer to effectively recognize the imaging timing of a performance having accompaniment.SOLUTION: A communication device 104 acquires from the outside both accompaniment sound data and reference data, including imaging timing information, related to a performance being performed based on a predetermined schedule. A microphone 103 inputs the sound of the performance. A digital signal processing circuit of a peripheral device control/processing unit 109 compares the input sound data of the microphone 103 and the accompaniment sound data included in the reference data. If both sound data match each other, a CPU 114 controls an image display control unit 107 to cause an image display unit 106 to display a time schedule which indicates on a time axis the degree of progress of the performance and the one or more imaging timing.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging support apparatus, an imaging apparatus, a communication terminal, and an imaging support control method. More specifically, the present invention relates to an imaging support apparatus, an imaging apparatus, and an imaging support apparatus that notify a photographer of shooting timing of a performance based on a certain schedule or program. The present invention relates to a communication terminal and an imaging support control method.

  Conventionally, an imaging apparatus having a function of notifying a user of a schedule or a program in advance is known. Patent Document 1 discloses a technique in which when an imaging apparatus approaches an IC tag installed at a person or a specific place, the imaging apparatus reads an installation position or a scheduled program start time from the IC tag and notifies the photographer. Yes. Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique for notifying a photographer in advance of a schedule and prompting photographing.

JP 2003-289265 A JP 2008-061195 A

  None of the techniques described in Patent Documents 1 and 2 does not notify the user of the shooting timing in the sense of a photo opportunity in real time. In addition, the schedule or program is notified before starting, but it cannot cope with sudden changes in the schedule or program. In addition, it is difficult to synchronize the start of the program with the prior art, and it is impossible to notify the exact time of shooting timing in the program.

  When shooting a certain moment, such as sports, you can know the approximate start timing of the competition (for example, the performance start time of a certain performer) and prepare for the shooting, but even the detailed shooting timing in the competition program I do n’t know. The same is true for theater. For this reason, it may be inadvertent and miss the important shooting timing. In addition, if continuous shooting is continued before the shooting timing, the memory buffer may be full and shooting may not be possible at the important shooting timing. That is, even with the techniques described in Patent Documents 1 and 2, it is still impossible to avoid a situation in which the shooting timing originally intended to be missed is missed.

  In addition, when there is a sudden change in the schedule or program, the photographer often cannot cope with the change, and the pre-preparation is not completed and the shooting at an appropriate timing is missed.

  An object of the present invention is to provide an imaging support apparatus, an imaging apparatus, a communication terminal, and an imaging support control method that refer to information accompanying the progress of a competition or play and notify a user of a desired shooting timing in real time. .

  In order to achieve the above object, an imaging support apparatus according to the present invention is an imaging support apparatus that supports shooting at one or more shooting timings included in a performance that is executed based on a predetermined schedule. Voice input means for inputting voice at the time, communication means for obtaining reference data including accompaniment voice data of the performance from the outside, comparison means for comparing the voice input by the voice input means with the accompaniment voice data, And display means for displaying a time schedule indicating the progress of the performance and the one or more shooting timings on a time axis according to the comparison result of the comparison means.

  According to the present invention, it is possible to shoot without missing the shooting timing in the performance program. By appropriately updating the reference data, it is possible to easily cope with sudden changes in competition and performance schedules.

It is a schematic block diagram of one Example of this invention. It is a schematic block diagram of the part regarding the audio | voice comparison of a present Example. It is an operation | movement flowchart of the audio | voice comparison of a present Example. It is an example of a screen which designates an imaging target. It is an example of a screen when photographing figure skating. It is an example of a change of a time schedule.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows a schematic block diagram of an imaging apparatus incorporating an embodiment of an imaging support apparatus and an imaging support control method according to the present invention. The imaging apparatus 100 illustrated in FIG. 1 can capture both still images and moving images.

  The image pickup apparatus 100 includes an image pickup element 102 that converts an optical image obtained by a photographing optical system into an electric signal, a microphone 103 as a voice input unit, and a communication device 104 that communicates with an external device. The peripheral device control / processing unit 109 processes the output image signal of the image sensor 102 and the output audio signal of the microphone 103, and exchanges data and control signals with an external device via the communication device 104. The peripheral device control / processing unit 109 includes an A / D converter that digitizes an image signal output from the image sensor 102 and a camera signal processing unit that converts output image data of the A / D converter into a predetermined format. It has.

  The CPU 114 controls the entire imaging apparatus 100 according to a program stored in the ROM 111. The CPU 114 and the peripheral device control / processing unit 109 cooperate to process an input image signal from the image sensor 102 and an input audio signal from the microphone 103 in accordance with a program stored in the ROM 111. The CPU 114 uses the RAM 113 as a temporary storage area for image data, audio data, and external data. The peripheral device control / processing unit 109, the CPU 114, the RAM 113, and the ROM 111 are connected to the system bus 112. The peripheral device control / processing unit 109 and the CPU 114 can exchange control signals and data with each other via the system bus 112, and can access the RAM 113 and the ROM 111.

  The image display control unit 107 causes the image display unit 106 to display an image of the processed image data. The speaker 110 outputs sound data from the peripheral device control / processing unit 109 as sound. The recording medium 101 can be image data such as still images and moving images, audio data, and external data.

  The user can input various instructions to the peripheral device control / processing unit 109 using the device operation unit 108.

  The communication device 104 is means for communicating with an external device to transmit / receive various data and control signals. The communication method may be a LAN (Local Area Network) that can communicate only within a specific area, or a communication that can be performed over a wider area such as a public line used by a mobile phone or the like.

  FIG. 2 shows a schematic block diagram of a voice processing system built in the peripheral device control / processing unit 109.

  The analog signal processing unit 201 removes a range other than the audible frequency range from the output audio signal of the microphone 103 with a low-pass filter (LPF), then converts it into a digital signal with an A / D converter, and digitally converts the obtained audio data. The signal processing unit 202 is supplied. Reference audio data 203 included in reference data input from the outside via the communication device 104 is also input to the digital signal processing unit 202. The reference sound data 203 is sound (accompaniment sound) reproduced or output as an accompaniment during the performance program of interest, or a part (first part) thereof. The digital signal processing unit 202 is a comparison processing unit that compares the audio data from the analog signal processing unit 201 with the reference audio data 203. Since both audio data substantially match, it is possible to know to what point in the performance program the actual performance has progressed. Although details will be described later, when both audio data substantially match, the digital signal processing unit 202 or the CPU 114 causes the image display control unit 107 to include a time schedule including marks indicating shooting timing and current time on the performance program. Output information. The image display control unit 107 superimposes and displays the time schedule indicating the progress of the performance program and the scheduled shooting timing on the image display unit 106 according to the time schedule information from the digital signal processing unit 202 or the CPU 114. The comparison process in the digital signal processing unit 202 is for knowing the progress position of the performance program, and as long as the reference audio data 203 is used, the reference audio data 203 may be a part of the audio reproduced by the performance program of interest.

  FIG. 3 shows an operation flowchart of comparison between audio data input from the microphone 103 and reference audio data. A computer program that realizes the processing shown in FIG. 3 is stored in the ROM 111, and the CPU 114 reads the computer program into the RAM 113 and executes it, thereby realizing the processing shown in FIG.

  In S300, the CPU 114 receives schedule / program information as reference data from the external device via the communication device 104 and the peripheral device control / processing unit 109, and temporarily stores it in the RAM 113. The CPU 114 supplies the schedule / program information received by the RAM 113 to the digital signal processing unit 202 of the peripheral device control / processing unit 109. The digital signal processing unit 202 stores the input schedule / program information for reference. The program information includes time required from the start to the end of the performance program, shooting timing, and music information (particularly accompaniment audio data) used in the performance program. The schedule / program information includes, for example, a part suitable for instructing the photographer of the photographing timing in the music data reproduced in the performance of the figure skating.

  In step S <b> 301, the CPU 114 starts capturing peripheral audio data using the microphone 103. Here, the accompaniment sound reproduced at the performance of the subject to be photographed is captured. The output audio signal of the microphone 103 is input to the digital signal processing unit 202 via the analog signal processing unit 201. The digital signal processing unit 202 sequentially compares the accompaniment sound data input from the microphone 103 and the reference sound data of the schedule / program information. The digital signal processing unit 202 sequentially supplies the comparison results to the CPU 114 and the image display control unit 107.

  In S302, the CPU 114 determines whether or not the comparison result by the digital signal processing unit 202 indicates that the input audio data matches the reference audio data. If the match is indicated, the CPU 114 proceeds to S303, and if it does not match, the CPU 114 retries S302. That is, the CPU 114 stands by until the audio data input from the microphone 103 matches the reference audio data. By detecting the coincidence, it is possible to know to what point in time the performance of interest has progressed from the start time, that is, the progress of the performance, and accordingly, a time schedule synchronized with the progress of the performance is displayed on the image display unit 106. It becomes possible.

  In step S303, the CPU 114 controls the image display control unit 107 to cause the image display unit 106 to start displaying the time schedule of the performance program. The time schedule is displayed in time synchronization with the actual performance. A display example of the time schedule will be described later with reference to FIG.

  In S304, the CPU 114 determines whether or not the performance program of interest has ended. When the notable performance program ends, the CPU 114 ends the processing shown in FIG. For the next performance program, the CPU 114 executes the process shown in FIG. 3 again.

  FIG. 4 shows an example of a screen for specifying or selecting the shooting timing by the program type. The schedule / program information received as external information (reference information) includes photographing timing information determined in advance. There are various shooting timings depending on the performance program, but here we will explain the case of figure skating as an example. In the case of figure skating, there are jump, spin, and step as the main performance composition, and the shooting timing is often the moment of such performance. These performances are synchronized with the music used in the performance program, and it can be said that the time from the start of the performance is almost fixed. Such information can be grasped in advance and can be collected as reference information. The user extracts the timing of these performances from the schedule information of the performance program fetched from the organizer or performer's server or the like, and adds it to the schedule information as shooting timing. The shooting timing determined and selected in this way can be displayed on the time schedule displayed on the image display unit 106. The user selects in advance on the selection screen as shown in FIG. The performance type to be photographed can be changed before or during the start of the performance program.

  FIG. 5 shows a screen example of the image display unit 106 waiting for a photo opportunity. On the screen of the image display unit 106, a time schedule 502 of this performance program is displayed so as to be superimposed on the subject image 501 being performed according to the performance program. The subject image 501 may be generated for an electronic viewfinder from an image captured by the image sensor 102. Further, in the case of a configuration in which an image display device such as a liquid crystal display or an organic EL element is superimposed on the optical finder, the subject image 501 may be an optical image by the optical finder.

  In the time schedule 502, a current position indicator 504 indicating the current position and one or more shooting schedule indicators 505 each indicating the shooting timing are displayed at corresponding time positions on the time axis 503 moving from left to right. The current position indicator 504 and the shooting schedule indicator 505 are each realized as a mark or icon having a predetermined shape. The current position indicator 504 is updated on the time axis 503 as time passes. The display position of the current position indicator 504 is determined in consideration of the time required for voice comparison. In the example shown in FIG. 5, the current position indicator 504 is illustrated as a vertical line and an arrow mark pointing in the time progress direction. The shooting schedule indicator 505 is illustrated as a downward-pointing triangular mark. The shooting schedule indicator 505 indicates, for example, the timing of the performance selected on the screen shown in FIG. 4 among the jump, spin or step of the performance of the figure skating. In FIG. 5, the time schedule is arranged at the lower part of the screen of the image display unit 106, but of course, it may be arranged at the upper part of the screen, the right part of the screen, or the left part of the screen. You may arrange | position in the place out of the display area.

  When the voice acquired by the microphone 103 matches the reference voice, the current position indicator 504 (vertical bar and arrow) is displayed at the corresponding time position on the time axis 503. As time passes, the current position indicator 504 moves to the right along the time axis 503. When the head of the arrow of the current position indicator 504 touches the next shooting schedule indicator 505, the photographer determines the shooting timing from the current state of the subject. The time from the current position indicator 504 to the next shooting schedule indicator 505 (remaining time until a photo opportunity) may be additionally displayed.

  FIG. 6 shows a specific example of the displayed time schedule 502.

  FIG. 6A shows a state in which the entire performance program time schedule 502 is displayed before the performance starts. A plurality of shooting schedule indicators 505 indicating the shooting timing selected in FIG. 4 are displayed on the time schedule. On the right end, the total time (here, 6 minutes) 601 is displayed.

  FIG. 6B shows an example of a partial enlarged display of the time schedule 502 immediately after the performance is started and the coincidence between the surrounding sound and the reference sound is detected. At this stage, the current position on the time schedule 502 is determined by comparing the acquired voice by the microphone 103 with the reference voice. At this time, the CPU 114 displays the current position indicator 504 so as to overlap the corresponding time position on the time axis 503 of the time schedule 502 as shown in FIG. In order to clarify details, the CPU 114 enlarges and displays the time of interest in the time schedule 502, and shows an enlargement ratio 602 at the right end of the time axis. Here, the time axis is enlarged three times, and the character “1/3” is additionally displayed as the enlargement ratio 602. In addition, a time (remaining time) 603 required to reach the shooting schedule indicator 505 is displayed above the shooting schedule indicator 505 immediately after. Here, “remaining 30 seconds” is displayed as the remaining time 603. It is possible to switch between the enlarged display shown in FIG. 6B and the entire display shown in FIG.

  FIG. 6C shows a display example of partial enlarged display when the current shooting time overlaps with the first shooting schedule indicator 505 and the two shooting schedule indicators 505 are located close to each other on the time axis. In the display example shown in FIG. 6C, the time axis is reduced to twice that of FIG. 6B, and the enlarged display is 3/2 times that of the whole display. An enlargement ratio 604 (here, “2/3”) is additionally displayed on the right end of the time axis. A time (remaining time) 605 required to reach the shooting schedule indicator 505 is displayed above the second shooting schedule indicator 505. Here, “remaining 40 seconds” is displayed as the remaining time 605.

  FIG. 6D shows a time axis from the middle to the end of the performance program. The display 606 at the right end (here, “3/3”) indicates that the display is up to the end of the program. “Remaining 20 seconds” is displayed as the remaining time 607 until the last shooting schedule indicator 505.

  The reference information is obtained from an external device by a method as described below, for example. The user acquires schedule / program information describing the accompaniment of the desired performance program and the content and timing of the performance from the server of the organizer or performer, and transmits it to the imaging apparatus 100. When it is necessary to add shooting timing or the like to the acquired schedule / program information, the user adds these before transmission to the imaging apparatus 100 so that they can be used as reference information. When the imaging device enters a specific area, necessary reference information may be automatically acquired from the server.

  A voice comparison for determining the degree of progress of the performance program to be noticed may be executed by a processing device outside the imaging device 100, and the comparison result may be transmitted to the imaging device 100. That is, the function of the digital signal processing unit 202 described with reference to FIG. 2 may be assigned to a server outside the imaging apparatus 100. For example, a voice recognition / control system capable of performing various operations by voice using voice recognition technology has been introduced for smartphones. This system is used for voice comparison of the present embodiment, and the comparison result is used as an imaging device. To receive.

  In the second embodiment, a function of notifying the user immediately before the shooting timing is added to the function of controlling the time schedule display so that the time schedule is displayed in synchronization with the progress of the performance program.

  An explanation will be given of athletics and motor sports. In the case of track and field competition, for example, when shooting timing is considered for the entire competition in medium- and long-distance running around a track, the final lap and goal scene are often the shooting timing. However, the time required from the start to the goal may deviate by several tens of seconds, and even if synchronization is established at the start of the competition, it is difficult to predict the exact goal time. Therefore, the external device determines the shooting timing based on the lap time just before the final lap, and the imaging device 100 receives the shooting timing synchronization signal from the external device when passing the point related to the final lap along with the shooting timing information. I tried to do it. In accordance with the shooting timing synchronization signal, the imaging apparatus 100 notifies the photographer of a photo opportunity by voice or a screen, and the photographer performs a shutter operation with reference to this notification. With such a configuration, it is possible to notify the user of the remaining time until the goal moment as an accurate shooting timing.

  Similarly, in motor sports, it is difficult to predict the goal timing at the start of the competition. The imaging timing synchronization signal determined from the lap time immediately before the final lap and the timing at which the point passing through the final lap is passed are transmitted from the external device to the imaging device 100, and the imaging device notifies the user of the photo opportunity. This makes it possible to notify the remaining time until the goal moment as an accurate shooting timing.

  In the above description, the goal point is the shooting timing, but not only the goal point but also a specific shooting point can be notified in the same way.

  As described above, even when the program is in progress, the shooting timing can be more accurately notified by using the system that obtains the shooting timing information and the synchronization signal of the shooting timing as external information at a specific point.

  The control of the CPU 114 may be performed by one hardware, or the entire apparatus may be controlled by a plurality of hardware sharing the processing.

  Although the present invention has been described in detail on the basis of preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. It is. Furthermore, each embodiment mentioned above shows only one embodiment of this invention, and it is also possible to combine each embodiment suitably.

  In the above-described embodiment, the case where the present invention is applied to a single imaging device, that is, a so-called digital camera has been described as an example. However, the present invention is not limited to this example, and an apparatus having an imaging unit and an audio input unit. If so, it is applicable. That is, the present invention can also be applied to a communication terminal, an information processing apparatus, a personal computer, or the like that includes an imaging unit and a voice input unit.

  The object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to the apparatus. At this time, the computer (or CPU or MPU) including the control unit of the supplied apparatus reads and executes the program code stored in the storage medium. The program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a magnetic disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, or a ROM can be used.

  In some cases, an OS (basic system or operating system) running on the apparatus performs part or all of the processing based on the above-described program code instructions, and the functions of the above-described embodiments are realized by the processing. included.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As mentioned above, although the preferable Example of this invention was described, this invention is not limited to these Examples, A various deformation | transformation and change are possible within the range of the summary.

Claims (10)

An imaging support apparatus that supports shooting at one or more shooting timings included in a performance performed based on a predetermined schedule,
Voice input means for inputting voice at the performance;
Communication means for acquiring reference data including accompaniment audio data of the performance from the outside;
Comparison means for comparing the voice input by the voice input means with the accompaniment voice data;
An imaging support apparatus, comprising: a display unit configured to display a time schedule indicating the progress of the performance and the one or more shooting timings on a time axis according to a comparison result of the comparison unit.   The imaging support apparatus according to claim 1, wherein the display unit updates the degree of progress of the time schedule on the time axis as time elapses.   The imaging support apparatus according to claim 1, wherein the reference data includes schedule information and program information regarding the performance.   The imaging support apparatus according to claim 1, wherein the display unit displays the time schedule together with the subject image of the performance.   5. The imaging support apparatus according to claim 1, wherein the time schedule includes a display of a remaining time from the present time to a time immediately after the photographing timing. 6.   The display of the time schedule by the display means can be switched between a whole display including the first to last performance and a partial enlarged display in which a part is enlarged. The imaging support device according to item.   An imaging apparatus comprising the imaging support apparatus according to claim 1. An imaging apparatus comprising the voice input unit, the communication unit, and the display unit of the imaging support apparatus according to claim 1,
An imaging apparatus, wherein the comparison result of the comparison means is acquired from the outside by the communication means. A communication terminal,
An imaging support apparatus according to claim 1;
Imaging means;
Operating means for instructing imaging by the imaging means;
A communication terminal comprising recording means for recording an image picked up by the image pickup means on a recording medium. A control method for supporting shooting at one or more shooting timings included in a performance performed based on a predetermined schedule,
A communication step of obtaining reference data including accompaniment voice data of the performance from the outside;
A comparison step of comparing the performance voice input by the voice input means with the accompaniment voice data;
And a step of displaying, on a display unit, a time schedule indicating the progress of the performance and the one or more shooting timings on a time axis according to a comparison result of the comparison step.

Images