JP7233886B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing apparatus and the like for photographing and video monitoring.

In recent years, as internationalization advances, many tourists are visiting Japan. Also, in competitive sports, opportunities for photographing athletes from various countries have increased significantly.

However, it is relatively difficult for both professional photographers and ordinary photographers to find a particular athlete from among a large number of athletes in, for example, a competitive sports scene. In addition, it is often the case that a plurality of athletes move quickly during a competition in a competitive sport, and the location of the athlete is often lost. The same applies not only to competitive sports, but also to the case of photographing or monitoring a specific person in a crowd.

Patent Literature 1 describes a plurality of cameras for photographing a subject from a plurality of directions, and a plurality of image processing devices for extracting predetermined regions from images captured by corresponding cameras among the plurality of cameras. Also, an image generation device is described that generates a virtual viewpoint image based on image data of a predetermined region extracted from images captured by the plurality of cameras by the plurality of image processing devices.
Patent Document 2 describes an automatic focus detection device that drives a focus lens based on an AF evaluation value obtained from a captured image and performs automatic focus detection control.

JP 2017-211828 A Patent No. 5322629

However, in a sports scene where many players are gathered, the players may be overlapped and lost. Also, it was difficult to shoot the athletes at the right time because they were out of the field of view.

Moreover, in particular, professional photographers need to immediately send the photographs they have taken to news bureaus or the like. Furthermore, even if I found a player I wanted to shoot, I had to focus on that player and then chase after the player I wanted to shoot. This is very difficult in fast-moving sports, and has the drawback of not being able to take good pictures if you are concentrating on this chasing.

In addition, on the server side, it is possible to grasp omnidirectional video and various information about the game and the field of competition, and it is possible to obtain various valuable information inside and outside the ground, but the conventional system does not make full use of the server. There was a problem that it was not
Similarly, there were many cases in which general users who were monitoring the game on stadiums or home terminals lost track of specific athletes or lost track of the game. Similarly, in car races, airplane races, horse races, and the like, there are times when a particular vehicle, airplane, horse, or the like is lost. Furthermore, when tracking a specific person on a street corner, the specific person may get lost in the crowd and be lost. In addition, there is a problem that if the user is concentrating on visually following such a specific object of interest, shooting, focusing, exposure adjustment, etc. for the object cannot be performed smoothly.
SUMMARY OF THE INVENTION It is an object of the present invention to solve the above-mentioned problems and to provide an image processing apparatus that makes it difficult for a photographer or an observer to lose sight of an object of interest, and enables smooth photographing, focus adjustment, exposure adjustment, etc. of the object. and

In the image processing device,
display means for displaying an image;
selection means for selecting a specific target from the screen displayed on the display means;
Designation information generation means for generating designation information regarding the specific target selected by the selection means;
transmitting means for transmitting the specified information generated by the specified information generating means to a server;
Acquisition means for acquiring from the server the location information of the specific target generated by the server based on the specified information;
a control means for controlling at least one of exposure adjustment and focus adjustment of the specific target based on the position information of the specific target acquired by the acquisition means ;
The control means displays the position of the specific target within or outside the display screen of the display means as additional information based on the position information.

According to the present invention, it is difficult for a photographer or an observer to lose sight of an object of interest, and focus adjustment and exposure adjustment for the object can be performed smoothly.

1 is a block diagram of an entire system including an image processing apparatus according to an embodiment; FIG. Server-side block diagram. Detailed block diagram of the terminal side. Detailed block diagram of the terminal side. The figure which shows an example of a player-of-interest display start sequence. The figure which shows an example of a player-of-interest display tracking sequence. The figure which shows the attention player display tracking control flow on the camera side. FIG. 10 is a diagram showing another example of the control flow for tracking and displaying the player of interest on the camera side; FIG. 3 is a block diagram showing a functional configuration example of a tracking unit 371 of the digital camera; The figure which shows the detection control flow of the player of interest on the server side. The figure which shows the flow which detects the uniform number of the player on the server side. The figure which shows the example of the attention player detection control flow including reserve. The figure which shows the example of a player-of-interest detection control flow. The figure which shows the example of a player-of-interest detection control flow. The figure which shows the AF display example of the attention player's camera display part. FIG. 10 is a diagram showing an example of a flow of attention player detection control and AF; FIG. 10 is a diagram showing an example of a flow of attention player detection control and AF; The figure which shows the example of a display of the camera display part at the time of automatic chasing. FIG. 11 is a diagram showing a display example of a camera display unit of a player of interest during automatic chasing; The figure which shows the example of the attention player detection control flow at the time of automatic chasing. The figure which shows the example of the attention player detection control flow at the time of automatic chasing. The figure which shows the example of an attention player change detection control flow. The figure which shows the example of an attention player change detection control flow. The figure which shows the example of a reserve player recognition control flow.

Modes for carrying out the present invention will be described below using examples.
First, an overview of a system using an image processing device for supporting shooting and video monitoring will be described with reference to FIG.

In Figure 1, the server (image processing server) side, which has multiple cameras for the server (fixed cameras and mobile cameras using drones, etc.), can determine the position of the player of interest (specific target) in the entire field of the stadium and the game. to grasp the latest situation in real time. Then, an example is shown in which the server timely provides necessary information, for example, for camera shooting or image monitoring, to the terminal held by each spectator.

In general, there are many places that cannot be seen or tracked by the angles and fields of view taken by professional photographers and general photographers. The same is true for spectators outside the stadium who are not filming. On the other hand, in the server-side system, it is possible to grasp and map in advance omnidirectional images and information on the entire game field (field coordinate information, etc.) based on images from multiple server cameras.
Therefore, the server grasps and distributes information that is invisible and difficult for each user to understand, so that the service to the audience can be greatly improved.

That is, multiple cameras (fixed cameras and moving cameras) for the server can track the position of each player, scores, fouls, judgment results of referees, and other up-to-date situations. Also, based on the information displayed on a large screen, etc., it can be analyzed by the server. As a result, the overall situation can be accurately recognized and timely transmitted to camera terminals, etc. owned by professional photographers and spectators, as well as terminals such as smartphones and tablets. As a result, the spectators can grasp the latest situation of the competition in a timely manner. In particular, professional photographers need to send the photos they have taken immediately to the news bureau, etc., but it is difficult to accurately grasp the overall situation of the competition just by looking at the camera screen because the field of view is narrow. However, by using the configuration of the present embodiment, the conditions of the game can be known quickly, so that it is possible to quickly select photographs to be sent to the news bureau or the like.

Terminals (image processing devices) used by professional photographers and spectators include a digital camera, a smart phone, a configuration in which a camera and a smart phone are connected, a tablet PC, a TV, and the like. A similar service can be provided to spectators who are watching the competition at home through a terminal (image processing device) such as a PC or TV through the Internet or television broadcasting, so that the situation of the competition can be grasped more accurately. You can enjoy the competition more.

In FIG. 1, 101 to 103 are cameras for servers, 101 (fixedly installed camera 1), 102 (fixedly installed camera 2), 103 (fixedly installed camera 3), 104 (large screen). , 110 (server), 111 (input means), and 112 (base station) acquire video and audio for providing information to general professional photographers and spectators. In this embodiment, there are three server cameras 101 to 103, but one or more cameras may be used. Also, these server cameras may be cameras mounted on, for example, drones instead of fixedly installed cameras. In addition to image acquisition and sound acquisition, input information other than image (for example, sound) can be taken in from the input means, and services for general professional photographers and spectators can be expanded.
Reference numeral 105 denotes a wired/wireless LAN or Internet, and 106 denotes a connection line for inputting information output from the input means 111 to the server 110 . Reference numeral 107 denotes a connection line for transmitting/receiving signals to/from the base station 112, and 108 denotes an antenna section for executing wireless communication of the base station.
That is, the blocks in the 100s are for supporting video shooting by professional photographers and general spectators.

On the other hand, in FIG. 1, 401 (terminal 1), 402 (terminal 2), and 403 (terminal 3) are terminals. It is a video display terminal device such as Here, 404 (antenna), 405 (antenna), and 406 (antenna) are antennas for wireless communication by 401 (terminal 1), 202 (terminal 2), and 203 (terminal 3), respectively.

When the server detects the position of the player of interest, for example, the terminal sends the ID information of the player of interest to the server side, and the server side sends various information such as the position information of the player to the terminal. Since the players are moving and the game situation is changing, it is necessary to detect the player of interest in a short period of time. Therefore, for wireless communication here, 5G or the like is used, for example.

Note that 401 (terminal 1), 402 (terminal 2), and 403 (terminal 3) may have a configuration in which a camera and a smart phone are connected and combined. In the lower right of FIG. 1, 301 is a smart phone, which mainly controls communication with the server. By installing application software on this smartphone, various video acquisition services can be realized. A digital camera 300 is an image processing device that is mainly used by professional photographers and spectators to take pictures and monitor images. Here, the camera 300 is connected to the smartphone 301 via USB, Bluetooth (registered trademark), or the like. An antenna 320 is used by the smartphone 301 to wirelessly communicate with the base station 112 .

If the terminal is a smartphone, etc., the exchange of video and control signals with the server is performed wirelessly. good. For example, if the wireless communication environment is 5G, communication is performed wirelessly, and if the wireless communication environment is LTE, information with a large amount of data is sent by wire, and control signals with a small amount of data are sent wirelessly. Furthermore, it is also possible to switch to wired communication depending on how busy the wireless communication line is.

Next, a detailed block configuration on the server side will be described with reference to FIG. In FIG. 2, the same reference numerals as in FIG. 1 indicate the same configuration, and the description thereof is omitted.
201 is an Ethernet (registered trademark) controller, and 204 is a detection unit that detects a play position according to a player's role (so-called position). Here, the roles (positions) of players are set in advance by registration or the like. For example, the roles (positions) in the case of rugby are 1 and 3 props, 2 hookers, 4 and 5 locks, 6 and 7 flanker, 8 number 8, 9 scrum half, 10 called a standoff. Also, 11 and 14 are called wings, 12 and 13 are called centers, and 15 is called a fullback.
As for where these players are, during set-pieces, the forwards are often in the attacking front and the backs are in the attacking back.

In other words, the player's position is generally determined according to the player's role (position), so it is better to follow the player after grasping the above-mentioned role (position) of the player of interest, and to chase the player more effectively and accurately. can be done.
Usually, the player's role can often be recognized from the uniform number. However, in some cases, the No. 10 player is injured, the No. 15 player stands off (enters the No. 10 position), the reserve player enters No. 15, and so on. Here, reserve players have uniform numbers from 16 to 23. However, the position is not decided only by the jersey number. Accordingly, the detection unit 204 detects the play position corresponding to the player's preset role, and information on the detected play position is taken into the CPU 211 in the server 110. There may be some changes due to player substitutions.

Reference numeral 205 denotes a contour information detection unit. For example, when a professional cameraman or spectator monitors an image on a terminal and takes a picture from that position and angle at the magnification of the camera, the server 110 detects the position of the player of interest. Notify the terminals 401 to 403 and the like. In addition, by notifying the terminals 401 to 403 of the profile information of the captured player of interest from the server 110, the terminals 401 to 403 can recognize the player of interest more reliably. The contour information detected by the block 205 is taken into the CPU 211 .

Reference numeral 206 denotes a player's face recognition unit, which finds the player from the video using AI, particularly image recognition technology such as Deep Learning, based on pre-registered face photo information of the player of interest. Information on the result of face recognition detected by the face recognition unit 206 is also taken into the CPU 211 .
A player physique recognition unit 207 finds a player based on pre-registered physique photo information of a noted player using the image recognition technology as described above.

Reference numeral 208 denotes a player's uniform number detection unit, which uses the above-described image recognition technology to find a player of interest from a number (such as uniform number) registered in advance. Needless to say, when detecting the player's number, not only the number on the back side of the bibs but also the number written on the front side may be detected. A position information generating unit 209 recognizes the position, direction, and angle of view of each camera from the position information of the cameras 101, 102, 103, etc. using GPS, etc., and the information on the direction and angle of view of the camera. Then, based on the images from each camera, the absolute positional information of the player on the ground is obtained by triangulation. The position information generating unit 209 also acquires positions in the screen such as poles and lines (for example, side lines and end lines) of the playing field that are set in advance in the stadium as reference indexes for detecting reference positions from the video in advance. You can leave it. Then, using those coordinates as reference coordinates, the absolute position of the player of interest in the stadium with respect to the field may be obtained.
210 is camera position information for detecting the position of each terminal and the direction and angle of view of the camera of each terminal from the position information, direction information, and angle of view information of each terminal sent from each terminal 401 to 403; This is the direction detector.

Reference numeral 211 denotes a CPU (Central Processing Unit) as a computer, which is a central processing unit that executes control shown in the following embodiments based on a control computer program stored in a program memory 712 as a storage medium. . It also serves as display control means, and controls information to be displayed on a display unit 214, which will be described later. A data memory 213 stores various data referred to by the CPU 211 . The data memory 213 stores information on past games, information on past players, information on today's game (competition), information on the number of spectators, information on the weather, information on featured players, current situations of players, and the like. . The information on the featured player includes information such as face, jersey number, and physique.
1101 is a data bus line within the server 110 .

Next, the details of the terminal side as the image processing device will be described with reference to FIGS. 3 and 4. FIG. FIG. 3 and FIG. 4 are block diagrams showing configuration examples of a terminal, and show the overall configuration of a camera 500 such as a digital camera as an example of a terminal using two drawings.
The digital camera shown in FIGS. 3 and 4 is capable of shooting moving images and still images, and recording shooting information. 3 and 4, a CPU (Central Processing Unit) 318, a program memory 319, and a data memory 320 are shown redundantly, but they are the same block and only one is built in each.
In FIG. 3, 301 is an Ethernet (registered trademark) controller. A storage medium 302 stores moving images and still images captured by a digital camera in a predetermined format.

Reference numeral 303 denotes an image sensor such as a CCD or CMOS, which converts an optical image from a light signal to an electric signal, further converts this information from analog information to digital data, and outputs the data. A signal processing unit 304 performs various corrections such as white balance correction and gamma correction on the digital data output from the image sensor 303 and outputs the data. A sensor driving unit 305 controls horizontal and vertical line driving for reading out information from the image sensor 303 and the timing at which the image sensor 303 outputs digital data.

Reference numeral 306 denotes operation unit input means. Inputs are made in response to selection and setting of various conditions for photographing with a digital camera, trigger operation for photographing, selection operation for using the flash, battery replacement operation, and the like. Also, in the operation unit input means 306, it is possible to select/set whether or not to perform AF (autofocus) on the player of interest based on the position information from the server. Selection/setting information as to whether or not to AF (autofocus) this player of interest is output from the operation unit input means 306 to the bus line 370 .
Furthermore, in the operation unit input means 306, it is possible to select/set whether or not to automatically track the player of interest based on the position information from the server. Information such as which player is to be designated as a player of interest (specific target) and whether or not automatic tracking of the player of interest is to be executed based on position information from the server is generated by the operation unit input means 306 as selection means. be. That is, the operation unit input means 306 functions as a designation information generation means for generating designation information regarding a specific object.

Reference numeral 307 denotes a wireless communication unit that functions as a transmission/reception unit, and is used for wireless communication between camera terminals possessed by professional photographers and general spectators with the server side. A magnification detection unit 308 detects the imaging magnification of the digital camera. Reference numeral 309 denotes an operation unit output unit for displaying UI information such as menus and setting information on an image display unit 380 that displays image information captured by a digital camera or the like. 310 is a compression/decompression circuit. After the digital data (RAW data) from the image sensor 303 is developed by the signal processing unit 304, it is compressed by the compression/decompression circuit 310 into a JPEG image file or HEIF image file, or RAW data is compressed as it is to form a RAW image file. On the other hand, when a RAW image file is developed within the camera to generate a JPEG image file or HEIF image file, processing is performed to decompress the compressed information and return it to RAW data.

Reference numeral 311 denotes a face recognition unit, which refers to face photograph information registered in advance in the server for a player of interest, and finds the player in the video by image recognition using AI, particularly, technology such as Deep Learning. is. Information on the result of face recognition detected by face recognition section 311 is taken into CPU 318 via bus line 370 .

A physique recognition unit 312 refers to the physique photo information registered in advance in the server regarding the player of interest, and finds the player of interest from the video using the above-described image recognition technology.
Reference numeral 313 denotes a player's uniform number detection unit, which finds a player from the uniform number of the player of interest (of course, the number on the front side may also be used) by the above-described image recognition technology. A direction detection unit 314 detects the direction in which the lens of the terminal faces. A position detection unit 315 detects the position information of the terminal using, for example, GPS.

A power management unit 316 detects the state of the power supply of the terminal, and supplies power to the entire terminal when it detects that the power button is pressed while the power switch is off. Reference numeral 318 denotes a CPU as a computer, which executes control shown in the following embodiments based on a computer program for control stored in a program memory 319 as a storage medium. It also serves as display control means and controls image information displayed on the image display section 380 . Note that the image display unit 380 is a display unit using liquid crystal, organic EL, or the like.
The data memory 320 is for storing the setting conditions of the digital camera, photographed still images, moving images, and attribute information of the still images and moving images.

In FIG. 4, 350 is a photographing lens unit which has a fixed 1st group lens 351 , a zoom lens 352 , an aperture 355 , a fixed 3rd group lens 358 , a focus lens 359 , a zoom motor 353 , an aperture motor 356 and a focus motor 360 . A fixed 1st group lens 351, a zoom lens 352, an aperture 355, a fixed 3rd group lens 358, and a focus lens 359 constitute a photographing optical system. For convenience, each of the lenses 351, 352, 358, and 359 is illustrated as one lens, but each may be composed of a plurality of lenses. Also, the photographing lens unit 350 may be configured as an interchangeable lens unit that can be attached to and detached from the digital camera.

A zoom control unit 354 controls the operation of the zoom motor 353 to change the focal length (angle of view) of the photographing lens unit 350 . A diaphragm control unit 357 controls the operation of a diaphragm motor 356 to change the aperture diameter of the diaphragm 355 .

The focus control unit 361 calculates the defocus amount and defocus direction of the photographing lens unit 350 based on the phase difference between the pair of focus detection signals (A image and B image) obtained from the image sensor 303 . The focus control unit 361 then converts the defocus amount and defocus direction into the drive amount and drive direction of the focus motor 360 . The focus control unit 361 controls the operation of the focus motor 360 based on this drive amount and drive direction, and drives the focus lens 359 to control the focus of the photographing lens unit 350 (focus adjustment). In this manner, the focus control unit 361 performs phase difference detection autofocus (AF). Note that the focus control unit 361 may perform contrast detection AF that searches for a contrast peak of the image signal obtained from the image sensor 303 .
371 is a tracking unit for tracking the player of interest with the digital camera itself. Tracking here means, for example, moving a frame display surrounding the player of interest within the screen, and focusing or adjusting the exposure on the player of interest tracked by the frame.

Next, an example of a player-of-interest display start sequence will be described with reference to FIG. This is the sequence performed by server 110 and camera 500 . FIG. 5A shows a sequence in which the server 110 responds to a question (request) from the camera 500. FIG. Then, the server 110 side provides the camera 500 side with information on the absolute position of the player of interest.
The camera 500 notifies the server 110 of attention player designation information (ID information such as uniform number and player name). At that time, the user may touch the position of the player of interest on the screen of the terminal, or may surround the player of interest with the finger while touching the screen with the finger. Alternatively, a list of multiple players may be displayed on the screen using the menu, and the name of the player of interest may be touched. Alternatively, a character input screen may be displayed on the screen and the player's name and jersey number may be entered. You may At this time, by touching the face position of the player of interest on the screen, the face may be image-recognized, or the player's uniform number may be image-recognized, and the player's name and uniform number may be sent. Alternatively, the face image itself may be sent to the server without image recognition, and image recognition may be performed on the server side. At this time, if there is a predetermined password, it is also sent to the server. On the server side, information regarding the absolute position of the player is sent to the camera based on the information specifying the player of interest (ID information such as uniform number and player name) by a block that supports image shooting. If a password is also sent from the camera, the contents of the information to be sent to the camera are changed accordingly.

Further, in FIG. 5A, the camera also sends information such as the position information of the camera used by the professional photographer and the audience, the direction of the camera, the magnification of the camera, and the like to the server. On the server side, a free-viewpoint image is created at the position and direction the camera is looking at, and the image seen by the actual camera is recognized from the magnification of the camera. The positional information indicating where the player is in the image actually seen by the camera, the player's contour information, etc. are sent to the camera. Based on the positional information, contour information, etc. sent from the server, the camera displays the player of interest more accurately and conspicuously on the screen of the display unit of the camera, and performs AF and AE on the player of interest.

An example of the sequence for starting the display of the player of interest for finding the player of interest has been briefly described, but there are many cases where it is desired to continue chasing the player. Therefore, next, a sequence for continuing tracking of the player of interest will be described with reference to FIG. 5(B). In FIG. 5B, the camera 500 as a terminal makes inquiries (requests) to the server 110 many times, for example periodically, to continuously recognize the player's position. In FIG. 5B, the camera periodically sends a player of interest display start sequence (A1, B1, . . . ) to the server, and the server periodically sends a player of interest display start sequence (A2, B2, ..). The operation of receiving and recognizing the position of the player of interest is repeated many times.

Further, FIG. 6 shows a method of automatically tracking the player of interest by the camera itself. The ID information of the player of interest is sent from the camera 500 to the server 110, and the position information of the player of interest is temporarily obtained from the server. After the position information is obtained and the position of the player of interest is narrowed down by referring to the position information, the camera 500 itself continues to track the player of interest by image recognition. In the focused player display tracking sequence of FIG. 6, the camera 500 itself tracks the focused player using image recognition technology. Re-request the location of the featured player. Specifically, when the camera loses sight of the player of interest, the camera sends the player of interest display start sequence (A1) to the server again, receives the player of interest display start sequence (B2) from the server, and displays the player of interest. Display the position on the screen. After that, the camera itself tracks the player of interest by image recognition again.

In this embodiment, an example of a service for assisting professional photographers and spectators in photographing is described, but the service may also be used for remote camera control. By sending this information from the server, it is possible to track the player and capture the decisive moment with the remote camera mounted on the automatic pan-tilt head.
Also, although this embodiment has been described using an example of shooting assist, the terminal may be a home TV. That is, when a spectator watching TV specifies a player of interest, the server may send the position information of the player of interest to the TV and display the player of interest conspicuously by displaying a frame or the like. Note that the player of interest may be indicated not only by the frame but also by a cursor (for example, an arrow), and the color and brightness of the region of the position of the player of interest may be made different from other portions. When the player of interest is outside the screen of the terminal, the direction of deviation from the screen of the terminal may be indicated by an arrow or text. Also, when you are outside the display screen, how far away from the current viewing angle the terminal is (is it deviated), how much the terminal must be rotated before the player of interest can enter the display screen, etc. , numbers or scales may be used.
Furthermore, when the player of interest is on the screen, the additional information is displayed on the screen, and when the player of interest moves out of the screen, the user is instructed not to indicate that the player is out of the screen with an arrow or the like. may be selected.
Alternatively, if the situation of the game is automatically determined and, for example, when the player of interest has left the bench, even if the player of interest moves off the screen, the fact that the player is off the screen may not be displayed with an arrow or the like. good. If the user can select a mode in which display of additional information is automatically erased or a mode in which display of additional information is not automatically erased, usability is further improved.

Next, an example of the control sequence on the camera side in FIG. 6 will be described with reference to FIGS. 7(A) and 7(B). FIGS. 7A and 7B show a player-of-interest display tracking control flow on the camera side.
In FIG. 7A, S101 represents initialization. In S102, it is determined whether or not photography has been selected. If photography has been selected, the process proceeds to S103, and if photography has not been selected, the process proceeds to S101. In S103, camera setting information is obtained. In S104, it is determined whether or not shooting (designation) of the player of interest has been selected. If shooting of the player of interest has been selected, the process proceeds to S105, and if shooting of the player of interest has not been selected, the process proceeds to S110 and performs other processing. do. In step S105, information on the player of interest (ID information of the player of interest, etc.) and a password, if any, are sent from the camera to the server. As a result, the server side detects the position information of the player of interest and transmits it to the camera. In S106, the positional information of the player of interest is received from the server.

In S107, the camera itself tracks the player of interest while referring to the position information sent from the server. Here, for example, the camera itself performs image recognition and tracks the player of interest. In that case, the player is tracked based on the player's jersey number, the player's facial information, the player's physique, or any combination thereof. That is, the shape of a part or the whole of the player of interest is image-recognized and tracked. However, if the user's shooting position is bad, the camera's field of view is narrow, or depending on the shooting angle, it may be hidden behind other subjects, so it may be lost. Send a location request. S107-2 shows an example of mark display as additional information for the player of interest. That is, as the additional information, a cursor indicating the player of interest is displayed, a frame is displayed at the position of the player of interest, the color or brightness of the position of the player of interest is changed so as to stand out, or a combination thereof is displayed. Characters other than the mark may be displayed. Then, in a state in which a live view image from the image sensor is displayed on the image display unit, additional information indicating the position of the player of interest is superimposed.

FIG. 7B shows an example of the flow of S107-2 for displaying marks, which will be described later. It should be noted that the user may be able to select with a selection switch not to skip and not execute the tracking operation of S107 as described above. Alternatively, a mode may be provided in which the tracking action is performed when the player of interest is on the screen, but not when the player is out of the screen, and the mode can be selected. Furthermore, it automatically judges the situation of the competition, for example, when the player of interest enters the bench, automatically stops the tracking operation of the player of interest outside the screen (additional information such as arrows is displayed, etc.) may be controlled. Alternatively, if the server recognizes that the player of interest has entered the bench regardless of whether it is on the screen or off the screen, the display of the position of the player of interest on the screen, the autofocus on the player of interest, and the automatic exposure adjustment for the player of interest are stopped. may be controlled to

In S108, it is determined whether or not it is OK (successful) to continue tracking the player of interest. If it is not successful in continuing the tracking of the player of interest, the process proceeds to S109.
In S109, it is determined whether or not the shooting of the player of interest has been completed. If the shooting of the player of interest continues, the process advances to S105 to send the information of the player of interest to the server again, and in S106 the information of the player of interest is received from the server, and the position of the player of interest is recognized again. , Continue to shoot the featured player. That is, if the tracking fails, the result in S108 is No, and if the tracking is to be continued in that case, the process returns to S105 to request the location information from the server.
FIG. 7(B) shows an example of the flow for displaying the player mark of interest in S107-2 on the camera side. In S120, based on the position information received from the server, the relative position of the player of interest on the display is calculated and obtained. In S121, while the live view image from the image sensor is being displayed on the image display unit, a mark or the like indicating the position of the player of interest is superimposed.

In the above embodiment, the server 110 reads the image of the entire competition field, for example, and acquires the coordinates, so it can grasp from where the image is taken with respect to the competition field from the images taken by the professional cameraman and spectators. You can also In other words, the server grasps in advance an image of the entire playing field from a plurality of server cameras (fixed cameras and moving cameras). As a result, it becomes possible to map the absolute position information of the player of interest on the field and the video that professional photographers and spectators see on terminals and digital cameras.
Also, when terminals such as professional cameramen and spectators' cameras receive the absolute position information of the players from the server, it becomes possible to map the absolute position information and the video currently being shot or monitored. For example, let (X, Y) be the absolute position information of the player of interest within the field from the server. It is necessary to convert this absolute position information into relative position information (X', Y') when viewed from the camera according to the position information of each camera. The above conversion from absolute position information to relative position information may be performed on the camera side as in S120, or the relative position information may be sent to individual terminals (cameras, etc.) after conversion on the server side. .

When performing the above conversion on a terminal such as a camera, relative position information (X', Y '). This relative positional information is used as positional information within the display screen of the camera.
On the other hand, when the server performs the above conversion, the server converts the absolute position information (X, Y) into relative position information (X′, Y'). The server sends this relative position information to each camera, and the camera that receives this relative position information converts this relative position information into the position information within the display screen of each camera. and

As described above, it is possible to take a good picture of the player of interest without missing the timing, because the terminals such as cameras of professional photographers and spectators are less likely to lose sight of the player of interest.
FIG. 8 shows another example of the attention player display tracking control flow on the terminal side such as the camera. In FIG. 8, S101, S102, S103, S104, S105, S106, S107, S107-2, and S110 are the same control as in FIG. 7, and description thereof is omitted.

In S131 of FIG. 8, it is determined whether or not the continuation of tracking the player of interest is OK (successful). If not, the process proceeds to S132. In S132, it is determined whether or not the shooting of the player of interest has been completed. If the shooting of the player of interest continues, the process advances to S105, where the information of the player of interest is sent to the server again. Continuing to shoot notable athletes. In S133, it is determined whether or not the position of the player of interest from the server has been detected. If the position of the player of interest has been detected from the server, the process proceeds to S106. Proceed to S101. In S134, it is determined whether or not the position of the player of interest has been detected from the server. If the position of the player of interest has been detected from the server, the process proceeds to S106. proceed to

Next, the tracking unit 371 of the digital camera will be explained using FIG.
FIG. 9 is a block diagram showing a functional configuration example of the tracking unit 371 of the digital camera. The tracking unit 371 is composed of a matching unit 3710 , a feature extraction unit 3711 and a distance map generation unit 3712 . The feature extraction unit 3711 identifies the image area (subject area) to be tracked based on the position information sent from the server. Then, a feature amount is extracted from the image of this subject area. On the other hand, the collating unit 3710 refers to the extracted feature amount in the captured image of each frame continuously supplied, and searches for an area having a high degree of similarity with the object area of the previous frame as the object area. Note that the distance map generation unit 3712 can acquire distance information to the subject from a pair of parallax images (images A and B) from the imaging device, and the identification accuracy of the subject area in the matching unit 3710 can be increased. However, the distance map generator 3712 may not necessarily be provided.
When the matching unit 3710 searches for an area highly similar to the object area as the object area based on the feature amount of the object area in the image supplied from the feature extraction unit 3711, for example, template matching or histogram matching is used. .

Next, a control flow for detecting a player of interest on the server side will be described with reference to FIGS. 10 and 11. FIG.
Based on the ID information and the like of the player of interest sent from a terminal such as a camera, the server recognizes the image of the player of interest. The server detects the player's position information based on the images from multiple cameras for the server (fixed cameras, moving cameras, etc.), and sends the player's position information to the camera terminals of professional photographers and spectators. . In particular, when a professional cameraman or spectator takes a picture, if there is positional information of the player of interest from the server, the player of interest can be reliably photographed without mistaking the player. Furthermore, information from the server is also important when tracking a player of interest with a camera and losing sight of the player in a blind spot. On the server side, the player's position information is continuously detected based on the images from multiple cameras for the server.

FIG. 10 shows the main flow of control for detecting a player of interest on the server side.
In FIG. 10, initialization is first performed in S201. Next, in S202, it is determined whether or not photography has been selected in the camera, and if photography has been selected, the process advances to S203 to obtain camera setting information. At this time, if there is a password in the setting information of the camera, it is also obtained. If photography is not selected, the process proceeds to S201. In S204, it is determined whether or not shooting (designation) of the player of interest has been selected. If shooting of the player of interest has been selected, the process proceeds to S205. number). If shooting of the player of interest is not selected in S204, the process proceeds to S210 to perform other processing.

In S206, the server finds the player of interest on the screen by image recognition based on the images from a plurality of cameras (fixed cameras, moving cameras, etc.) based on the ID information of the player of interest. At S207, the server tracks the player of interest based on the images from the multiple cameras. In S208, it is determined whether or not it is OK (successful) to continue tracking the player of interest. Keep track. If it is determined in S208 that the player of interest has not been successfully continued to be tracked, the process proceeds to S209.

In S209, it is determined whether or not the shooting of the player of interest is finished. When the shooting of the player of interest is finished, the process returns to S201. Again, based on the ID information of the player of interest, the server searches for information from a plurality of server cameras (fixed cameras and moving cameras) to find the player of interest, and in S207 continues from the plurality of cameras. Based on the video, track the player of interest.

Next, an example of a method for finding a player of interest in S206 and tracking him in S207 will be described with reference to FIG.
FIG. 11 shows a target player detection control flow using uniform number information. In FIG. 11, in S401, the server obtains the uniform number from the data memory 213 based on the ID information of the player of interest, searches for this uniform number from the image information of a plurality of cameras for the server by image recognition, Get the position information of the number player. In S402, the absolute position information of the player of interest is obtained by integrating the position information obtained from the images of the plurality of cameras for the server. By synthesizing the information of a plurality of cameras for the server in this way, the accuracy of the absolute position information of a player with a certain uniform number is improved. In S403, the absolute position of the player of interest detected in S402 is transmitted to terminals such as cameras owned by professional photographers and spectators. In S404, it is determined whether or not the player of interest is being tracked. If the player of interest is being tracked, the process returns to S401, and if the player of interest is not being tracked, the flow of FIG.

In addition, using the video from at least one of the multiple cameras for the server, find the uniform number of the player of interest, the visible size, angle, and even the background (playing field) information By inputting , it is possible to acquire the position information of the player of interest. Also, using images from multiple cameras for the server, similarly, it is possible to find the uniform number of the player of interest, and input the visible size, angle, and background (field) information. Therefore, it is possible to improve the accuracy of the position information of the player of interest.

At this time, for example, in the case of rugby, for starting members with uniform numbers 1 to 15, the player's uniform number indicates the role of that player. Reserve players have jersey numbers 16 to 23, and in principle, they are players from forwards to backs in order from lower numbers to higher numbers. But depending on the team, I don't know how many forwards, how many backs, and reserves.
Therefore, if the positions are registered in the server in correspondence with the player information, the roles of players with jersey numbers 16 to 23 will be clarified.

As described above, the server has the data memory 213, and the data memory 213 stores a plurality of pieces of previously photographed face information of players registered as members of the game. Furthermore, the server has means for detecting the player's face information based on the images from the multiple server cameras. Based on the face information detected from the multiple server cameras, the server compares the face information taken in the past of the players registered as members in the game with multiple photos to detect the player of interest. . If roles (positions) are registered in the server in correspondence with player information, the roles of the players with uniform numbers 16 to 23 will also be clarified. It should be noted that AI may be used for face recognition by comparing videos obtained from a plurality of cameras with images of players registered as members of the game.

Also, when performing this face recognition, if we focus on the roles of the players, including the reserve players, we can reduce the time required to detect the players on the server. Accuracy can be improved.
FIG. 12 shows an example of a player detection control flow including reserve players in that case.

In FIG. 12, in S801, the server detects the positional information of the ball from the images of the multiple cameras. A rough position of the player of interest is estimated from the positional information of the ball. In addition, the roles of players such as forwards and backs (starting member players recognize the player's role by the uniform number. For reserve players, the already registered player information and the player's uniform number in today's game The name, jersey number, and role of the reserve player are recognized according to the player's role.), the area for searching the face information of the player is recognized.

In S802, for the area recognized in S801, the server recognizes the face information of the player of interest, including the reserve player, inputs video information from a plurality of cameras, and acquires the player's position information based on this face information. do.
When the images of each camera among multiple cameras find the face information of notable players including reserve players, by entering the visible size, angle, and background (field) information, It is possible to acquire the location information of notable players, including reserve players. Similarly, if you find the face information of a player of interest, including a reserve player, in the images of multiple cameras, you can input the size, angle, and background (field) information that you can see, The accuracy of the position information of the attention player including the player is improved.

In S803, the absolute position of the player of interest is detected based on the position of the player of interest from the image information of the plurality of cameras detected in S802. In S804, the absolute position of the player of interest detected in S803 is transmitted to camera terminals owned by professional photographers and general spectators.

In S805, it is determined whether or not the player of interest is being tracked. If the player of interest is being tracked, the process advances to S801.
Again, the game or competition situation (whether a team is attacking or defending) is determined according to the position of the ball. is not limited to For example, if one team commits a foul, the opposing team is given a penalty kick. In this case, the match is likely to move the team that gets the penalty kick forward from the current ball position. Therefore, it is possible to perform control based on the game situation in which this forward movement is predicted. In this way, the position of the ball may be predicted from the situation of the foul.

Information on player substitutions may be recognized based on images captured by a plurality of server cameras, and information on players leaving the field and players entering the field (including the positions of these players) may also be recognized. The server then sends this information to camera terminals owned by professional photographers and general spectators. Players who enter the field at the time of player substitution will follow where they will enter from, and at the same time, notify camera terminals owned by professional photographers and general spectators.
With reference to FIG. 13, a method of assisting player detection in player substitutions based on reserve player detection control will be described. FIG. 13 shows a control flow for detecting a player of interest including a reserve player using face recognition information in consideration of the role (position) of the player at the time of player substitution.

In FIG. 13, the same reference numerals as in FIG. 12 indicate the same steps, and description thereof will be omitted.
In S905, it is determined whether or not the player of interest is being tracked. If the player of interest is being tracked, the process advances to S906. In S906, it is determined whether or not the player has been changed. If the player has been changed, the process proceeds to S907. In S907, the server recognizes the player substitution and updates the player information list participating in the game.

Here, the game situation (whether a certain team is attacking or defending) is determined according to the position of the ball, but control based on the game situation is not limited to the position of the ball. For example, if one team commits a foul, the opposing team is given a penalty kick. In this case, the game is more likely that the team that gets the penalty kick will move forward from where the ball is now. Therefore, it is possible to perform control based on the game situation in which this forward movement is predicted. In this way, the position of the ball may be predicted from the situation of the foul. Furthermore, regarding the player change in S906, for example, the server may recognize and detect when a player leaves the field and when a player enters the field using a plurality of cameras. In addition to this, there is a method of recognizing and detecting the referee's call as voice information. There is also a method of detecting from player substitution information displayed on a large screen. Furthermore, you may make it detect from the member list displayed on the large screen.

When detecting a player of interest, it is also possible to detect information about a player who has committed a foul, such as "Sing Bing", and temporarily withdraws from the images captured by a plurality of server cameras. The server then sends this information to camera terminals owned by professional photographers and general spectators. A player who is deemed a shin-bin is out of the game for 10 minutes.
In the example of FIG. 13, information about player substitutions is recognized from images captured by a plurality of cameras, and information about players leaving the field and players entering the field (including the positions of these players) is recognized. The server then sends this information to camera terminals owned by professional photographers and general spectators. Players entering the field at the time of player substitution are chasing where they are entering from, and at the same time, they are notified to camera terminals owned by professional photographers and general spectators.

However, players leave the field for other occasions besides substitutions. When a player commits a foul, depending on the severity of the foul, a red card that calls for an immediate ejection, or a foul that denies the player from participating in the game for 10 minutes, means that the player leaves the field at least temporarily. Therefore, a method for assisting the detection of a player in a player substitution and the detection of a player who has committed a foul will be described with reference to FIG. FIG. 14 shows a player-of-interest detection control flow that considers the roles (positions) of players who are replaced or who leave the field at least temporarily.

In FIG. 14, in S1001, the server detects the positional information of the ball based on the images of the multiple cameras. A rough position of the player is estimated from the positional information of the ball. In addition, the roles of players such as forwards and backs (players in the starting members recognize the roles of players by their jersey numbers. Reserve players are the player information already registered and the player's jersey number in today's game and its The player's role recognizes the reserve player's name, jersey number, and role.), the area for searching the face information player is recognized. In S1002, the server recognizes a plurality of pieces of face information of the player of interest including the reserve player, and acquires the player's position information of this face information from the image information of the plurality of cameras. If you find the face information of a player of interest, including a reserve player, from the image of one camera out of multiple cameras, you can enter the size, angle, and background (field) information that you can see. , it is possible to acquire position information of notable players, including reserve players.

Similarly, if you find the face information of a player of interest, including a reserve player, from the images of multiple cameras, you can input the size, angle, and background (field) information that you can see. It is possible to acquire the position information of notable players including , with high accuracy. In S1003, based on this input information, the absolute position of the player of interest including the reserve player is detected.

In S1004, the absolute position of the player of interest detected in S1003 is transmitted to camera terminals owned by professional photographers and general spectators. In S1005, it is determined whether or not the player of interest is being tracked. If the player of interest is being tracked, the process advances to S1006. In S1006, it is determined whether or not a temporary exit foul has been committed. In S1007, it is determined whether or not the temporary dismissal was a red card foul, and if it is a red card foul, the process proceeds to S1008. The process proceeds to S1009 in the case of Shin Bin's foul. In S1008, the server recognizes the player who received the red card, excludes him from the members participating in the match, and updates the list of participating player information.

In S1009, the server recognizes the player who received the shinbin, removes the player from the players participating in the game for about 10 minutes (10 minutes is a guideline), and updates the player information list. Here, it recognizes that the player who received the shinbin returns to the field, and updates the participating player information list. Here, the player's role = position and the position of the ball determine the situation of the game (whether a certain team is attacking or defending). Control is not limited to ball position.

For example, if one team commits a foul, the opposing team is given a penalty kick. In this case, the game is more likely that the team that gets the penalty kick will move forward from where the ball is now. Therefore, it is possible to perform control based on the game situation in which this forward movement is predicted. You may predict the position of the ball from the fact that it is a foul.

Furthermore, with regard to the foul of at least temporary exit in S1006, for example, the server may recognize and detect the fact that the player commits a foul and leaves the field using a plurality of cameras.
Other than that, there is a method of recognizing and detecting the referee's call as voice information. Further, it may be detected from foul information displayed on a large screen.

In this embodiment, a player of interest is registered in advance, a mark indicating where the player of interest is is displayed on the display unit of the camera, and the focus of the player of interest is automatically adjusted (AF). . As a result, professional photographers and general spectators have the advantage of being able to quickly photograph the player of interest.

FIG. 15 shows the display unit on the camera side in AF (autofocus) for the player of interest. In FIG. 15, the attention player is registered as the player who is handing off in the middle of FIG. 15(A). The camera performs AF (autofocus) on the player of interest. The image viewed by the photographer on the display unit of the camera is shown in FIG. 15B, and since AF (autofocus) is applied to the player of interest, it is possible to shoot without missing a photo opportunity. At this time, exposure adjustment may be automatically performed for the player of interest.

Next, FIGS. 16 and 17 show the flow of AF for the player of interest in player-of-interest display tracking control on the camera side. The same reference numerals as those in FIG. 7 indicate the same steps, and the description thereof is omitted.
In FIG. 16, in S3807, the player of interest is tracked by the camera itself based on the position information of the player of interest. At that time, after marking the player of interest on the display unit, AF (autofocus) is performed on the player of interest. In S3808, it is determined whether or not the player of interest has been successfully continued to be tracked. If the continuation of tracking has not succeeded, the process proceeds to S109.

FIG. 17 shows details of the flow of S3807. In FIG. 17, in S3811, the camera receives the absolute position information of the player of interest from the server. In S3812, the camera converts the absolute position information of the athlete of interest into relative position information based on the position, direction, magnification, etc., captured by this camera. In S3813, information on the player of interest is displayed on the display unit based on the information on the relative position viewed from the camera. In S3814, information is input from the operation unit input means 906, and it is determined whether or not a mode for executing AF (autofocus) of the player of interest based on position information from the server has been selected. If the AF (autofocus) of the player of interest has been selected, the process proceeds to S3815, and if the AF (autofocus) of the player of interest has not been selected, the process proceeds to S3816. Note that when AF (autofocus) for the player of interest is not selected, AF or AE is performed in accordance with the frame displayed in the center of the display screen of the camera, regardless of the positional information of the player of interest. shall be

A well-known method may be applied to the AF (autofocus) method in S3815, and description thereof will be omitted. Also, in S3815, the exposure adjustment for the player of interest may be performed. In S3816, it is determined whether or not the player of interest is being tracked. If the player of interest is being tracked, the flow advances to S3811.
With the above control, camera terminals of professional photographers and general spectators can not only recognize the player of interest, but also quickly perform AF and AE on the player of interest, enabling timely photographing.

It should be noted that it is also possible to provide means for selecting the player-of-interest automatic chasing mode on the camera side. Here, if the player of interest automatic chasing mode is selected, the camera automatically zooms in on the player of interest on the screen of the display unit. Therefore, it is easier to use for professional photographers and general spectators.
FIG. 18 shows a display example of the camera display section during automatic chasing.
In FIG. 18, 3901 represents the display section of the camera. In 3901, seven athletes A, B, C, D, E, F, G, and H are in the shooting area for the cameras of professional photographers and general spectators. Here, the player of interest is K, but he is out of the shooting area of the camera.

3902 represents the zoomed-out state of the camera display unit when the automatic follow-up mode is ON. The camera is automatically set to the wide-angle side by the zoom function, and is controlled so that the player K of interest also enters the shooting area. FIG. 19 is a diagram showing a more specific display example. In FIG. 19A, an arrow is superimposed to indicate that the player of interest is outside the display screen while the live view image from the image sensor is being displayed. indicated by the direction of the arrow. FIG. 19(B) shows a case where the zoom has become wide due to the automatic follow-up mode. Here, the position of the attention player is also displayed with an arrow on the screen. In this way, the player of interest can be placed within the display screen, so that the situation of the game can be easily grasped, and an image that the user wants to take can be easily obtained.
FIG. 20 and FIG. 21 show the player-of-interest display tracking control flow on the camera side, that is, the flow in the automatic chasing mode for the player of interest.
In FIG. 20, the same reference numerals as in FIG. 7 indicate the same steps, and the description thereof is omitted. In FIG. 20, in S4007, the camera itself tracks the player of interest. Note that when the automatic chasing mode is selected by the operation unit, automatic chasing of the player of interest is performed, and when it is not selected, automatic chasing is not performed. When there are no athletes within the camera area, the camera automatically zooms out so that the athletes are within the camera display screen. Control. Details of the control in S4007 are shown in FIG. 21 and will be described later.
In S4008, it is determined whether or not the player of interest has been successfully continued to be tracked (OK or not). is executed, and if the continuation of the tracking of the player of interest has not succeeded, the process proceeds to S109.

Next, the details of S4007 will be described with reference to FIG. In FIG. 21, in S4011, the camera receives the absolute position information of the player of interest from the server. In S4012, the camera converts the absolute position information of the player of interest into relative position information based on the position, direction, magnification, etc. captured by this camera. In S4013, based on the information on the relative position seen from the camera, information on the player of interest is displayed on the display unit. In S4014, it is determined whether or not the player of interest is outside the area being photographed by the camera. If the player of interest is outside the area captured by the camera (outside the display image), the process proceeds to S4015, and if the player of interest is inside the area captured by the camera (within the display image), the process proceeds to S4018. In S4015, information is input from the operation unit input means 906, and it is determined whether or not the user has selected the automatic chasing mode for the player of interest. If the automatic chasing mode for the player of interest has been selected, the process proceeds to S4016, and if the automatic chasing mode for the player of interest has not been selected, the process proceeds to S4018. In S4016, the camera is zoomed out to the wide angle side until the player of interest is displayed on the display section of the camera. In S4017, AF (autofocus) of the player of interest is executed. At this time, AE is also executed so that the player of interest is properly exposed. In S4018, it is determined whether or not the player of interest is being tracked. If the player of interest is being tracked, the process advances to S4011.

As described above, the server reads the video of the entire competition field, and from the videos taken by professional photographers and general spectators, it is possible to ascertain where the video was taken. By capturing the entire field image from multiple cameras, the server will be able to map the position information of the field and the images seen by professional photographers and general spectators. Also, when professional photographers and cameras of general spectators receive the absolute position information of the athletes of interest from the server, it becomes possible to map the absolute position information and the image currently being shot. In other words, professional photographers and cameras of general spectators can recognize the athletes of interest and take pictures in a timely manner. Here, when the player of interest is not in the area photographed by the camera, the zoom is set to the wide-angle side and controlled so that the player of interest enters the area photographed by the camera. Furthermore, since the camera automatically adjusts the focus and exposure to the player of interest, professional photographers and the cameras of general spectators can quickly and reliably shoot images that are in focus on the player of interest.
Furthermore, in addition to AF (autofocus), AE (automatic exposure) is automatically executed, so an optimum image can be obtained without waiting for user's adjustment. Note that only AE may be performed without AF. Further, the user may selectively turn off control of one of AF and AE using a selection switch (not shown).

As described above, in sports such as rugby and soccer, it is very difficult to keep chasing a player of interest because players dodge opponents by taking steps that are difficult to predict. According to this embodiment, even in such a case, it is possible to continue chasing, or even if the attention player is lost, there is an advantage that it is possible to re-detect/re-track immediately. Furthermore, in sports such as rugby and soccer, the players with the ball change hands one after another. At that time, the notable athletes that the cameraman wants to shoot will also be replaced one after another. In this case, for example, there is also a method of chasing the player in possession of the ball.

The server may keep track of what's going on in the field and try to predict what might happen next. The server then notifies the camera terminals owned by professional photographers and general spectators of this prediction information. This prediction information is displayed on camera terminals owned by professional photographers and general spectators. Professional photographers and general spectators can more reliably obtain photo opportunities by viewing this information.
As a specific example of the prediction function, the case of prefetching player substitutions will be described.

The server judges (analyzes) the situation of the game (competition situation) with multiple cameras for the server, predicts what will happen next, and the information based on this is held by professional photographers and general spectators. Make a call to the camera terminal. In rugby, the possibility of player substitution increases when a player is injured. FIG. 22 shows a control flow for detection of player change of interest in which the player change timing is predicted from the reserve player's preparation status.

In FIG. 22, the same reference numerals as in FIG. 10 are the same steps, so description thereof will be omitted. In S4107, the player of interest is tracked. A specific flow will be described later with reference to FIG. At 4108, reserve players are recognized. Details of S4108 will be described later with reference to FIG. In S4109, it is determined whether or not the tracking of the player of interest continues. If the tracking of the player of interest continues, the process advances to S4107. In S4110, it is determined whether or not the shooting of the player of interest has ended. In S4111, it is determined whether or not the reserve player has moved. If the reserve player has moved, the process proceeds to S201, and if the reserve player has not moved, the process proceeds to S4108.

Next, the flow of S4107 will be explained using FIG.
In this example, it is assumed that the player himself incorporates the position sensor into his uniform or other clothing, or the player wears the position sensor on his arm, waist, or leg using a belt or the like. Then, by sending the information from this position sensor to the server side wirelessly by communication means, the server recognizes the signal from the player's position sensor, generates position information, and sends it from the server to the cameras owned by professional photographers and spectators. terminal.
In FIG. 23, in S4201, the server acquires the position sensor information of the player of interest from multiple cameras. Each of the plurality of cameras includes detection means for receiving radio waves from the position sensor, detecting the direction of the received radio waves and the level of the received radio waves, and outputting them as position sensor information. there is In S4202, the absolute position of the player of interest is detected from position sensor information from a plurality of cameras. In S4203, information on the absolute position of the player of interest is transmitted to camera terminals owned by professional photographers and general spectators. In S4204, it is determined whether or not the player of interest is injured. If the player of interest is not injured, the process proceeds to S4205. In S4205, it is determined whether or not the player of interest is being tracked. If the player of interest is being tracked, the process advances to S4201.

FIG. 24 shows the reserve player recognition control flow of S4108.
In FIG. 24, in S4301, the server acquires the position sensor information of the reserve player using a plurality of cameras. The position sensor information also includes the direction of the received radio waves and the level of the received radio waves. In S4302, the absolute position of the reserve player is detected from position sensor information of a plurality of cameras. In S4303, attention is paid to the movement of the reserve player. In particular, if the player to watch is a reserve, pay attention to its movement. In S4304, it is determined whether or not the reserve player has moved. If the reserve player has moved, the flow in FIG. 24 is terminated, and if the reserve player has not moved, the process advances to S4301.

As a play prediction, if you can respond to interceptions, etc., you will be able to ensure the photo opportunity. In other words, if professional photographers and general spectators can take pictures in response to unexpected player movements, they will be able to take valuable pictures.
Also, in baseball, for example, a change of players may be predicted based on statistical data, such as when a ball is hit, the possibility of a change of pitcher increases.
A player's movement may be stored as big data in a server, and AI may predict the player's movement based on this big data.
In the above description, the example of the attention player is one, but the attention player may be multiple players. In addition, it is assumed that the player to be noticed can be switched in the middle.
Further, in the above description, images include not only moving images but also still images.

In the above embodiments, the position of the player of interest can be displayed in a timely manner on the terminal side of the camera or the like. be able to.
It should be noted that the designation of the player to be noticed can be switched in the middle. All the players who are participating in the game may be selected as featured players. Also, video and images include not only moving images but also still images. In addition, I have mainly explained about chasing and tracking the attention player. However, instead of chasing only the player of interest, information about the player who has the ball or who receives the ball may be transmitted and displayed to the professional cameraman or spectators. Also, in the embodiment, an example of tracking a rugby player or the like has been described, but other sports players may be used, and the system may be applied to a system that tracks a person such as a specific criminal using a plurality of surveillance cameras. It goes without saying that we can. Alternatively, the present invention can be applied not only to a system for tracking a specific vehicle in car racing or the like, but also to a system for tracking a horse in a horse race or the like. In the embodiment, an example in which a player of interest is specified using a camera terminal or the like has been described, but the player of interest may be specified on the server side.

Also, for example, in international competitions such as the Olympics and World Cups, privileges are often given to some spectators and sponsors, etc. In the embodiment, depending on such privileges and contract levels, added value You can change the offer level. These levels of control can be realized by entering passwords, etc., and professional photographers with special contracts can acquire valuable footage and various information on and off the ground by entering passwords, enabling better photography. become.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.
Further, a computer program that implements the functions of the above-described embodiments for part or all of the control in the present invention may be supplied to the imaging device or the information processing device via a network or various storage media. Then, the computer (or CPU, MPU, etc.) in the imaging device or information processing device may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

101, 102, 103 (for servers) cameras 401, 402, 403 terminal 110 server 371 tracking unit 380 image display unit

Claims (30)

display means for displaying an image;
selection means for selecting a specific target from the screen displayed on the display means;
Designation information generation means for generating designation information regarding the specific target selected by the selection means;
transmitting means for transmitting the specified information generated by the specified information generating means to a server;
Acquisition means for acquiring from the server the location information of the specific target generated by the server based on the specified information;
a control means for controlling at least one of exposure adjustment and focus adjustment of the specific target based on the position information of the specific target acquired by the acquisition means ;
The image processing apparatus, wherein the control means displays the position of the specific object within or outside the display screen of the display means as additional information based on the position information. 2. The image according to claim 1 , wherein the server recognizes the specific target in the video, generates position information of the specific target based on the recognition result, and transmits the position information to the image processing device. processing equipment. 3. The image processing apparatus according to claim 1 , wherein the server generates position information based on a result of image recognition of the specific target and transmits the position information to the image processing apparatus. display means for displaying an image;
selection means for selecting a specific target from the screen displayed on the display means;
Designation information generation means for generating designation information regarding the specific target selected by the selection means;
transmitting means for transmitting the specified information generated by the specified information generating means to a server;
Acquisition means for acquiring from the server the location information of the specific target generated by the server based on the specified information;
a control means for controlling at least one of exposure adjustment and focus adjustment of the specific target based on the position information of the specific target acquired by the acquisition means;
The image processing apparatus, wherein the server recognizes a signal from a position sensor of the specific target and generates position information of the specific target based on the recognition result. 2. The image processing apparatus according to claim 1 , wherein said additional information includes at least one of a frame, a cursor, and areas of different colors and brightness. 6. The image processing apparatus according to claim 1 , wherein said additional information indicates in which direction said specific object is when viewed from the screen when said specific object is outside the screen. 7. The image processing apparatus according to claim 5 , wherein said additional information indicates how far said specific object is out of the screen. 8. The image processing apparatus according to claim 7 , wherein said additional information indicates how much said specific object is out of the screen by the length or thickness of an arrow. 8. The image processing apparatus according to claim 7 , wherein said additional information indicates how much said specific object is out of the screen by a number or a scale. 4. The image processing apparatus according to claim 3 , wherein the server performs image recognition of the number worn by the specific target and the shape of a part or the whole of the specific target. The server performs image recognition of the specific target in images from a plurality of cameras, generates position information of the specific target based on the recognition result, and transmits the position information to the image processing device. The image processing apparatus according to any one of claims 1 to 10 . 2. The image according to any one of claims 1 to 11 , further comprising tracking means for tracking the specific target after the position information of the specific target is obtained by the obtaining means. processing equipment. display means for displaying an image;
selection means for selecting a specific target from the screen displayed on the display means;
Designation information generation means for generating designation information regarding the specific target selected by the selection means;
transmitting means for transmitting the specified information generated by the specified information generating means to a server;
Acquisition means for acquiring from the server the location information of the specific target generated by the server based on the specified information;
Control means for controlling at least one of exposure adjustment and focus adjustment of the specific target based on the position information of the specific target acquired by the acquisition means;
tracking means for tracking the specific target after the position information of the specific target is obtained by the obtaining means;
The tracking means executes tracking of the specific target after acquiring the position information of the specific target from the server, and requests transmission of the position information to the server when tracking fails. Image processing device. 14. The image processing apparatus according to any one of claims 1 to 13 , wherein the server acquires in advance an image of an entire field in which the specific object exists and generates the position information. 15. The server according to claim 14 , wherein said server generates relative position information when said specific object is viewed from said image processing device based on position information of said specific object in said field. Image processing device. The server transmits first positional information of the specific object in the field to the image processing device, and the image processing device views the specific object from the image processing device based on the first positional information. 15. The image processing apparatus according to claim 14 , wherein the relative position information is generated. 17. The image processing apparatus according to any one of claims 1 to 16 , wherein said selection means selects a plurality of specific objects. display means for displaying an image;
selection means for selecting a specific target from the screen displayed on the display means;
Designation information generation means for generating designation information regarding the specific target selected by the selection means;
transmitting means for transmitting the specified information generated by the specified information generating means to a server;
Acquisition means for acquiring from the server the location information of the specific target generated by the server based on the specified information;
a control means for controlling at least one of exposure adjustment and focus adjustment of the specific target based on the position information of the specific target acquired by the acquisition means;
The image processing, wherein the specific target is a specific player in a predetermined game, and an estimation means is provided for estimating the position of the specific player based on a preset role of the specific player. Device. 19. The image processing apparatus according to claim 18 , wherein the estimation means estimates the position of the specific player by also referring to the role of the reserve player. 20. The image processing apparatus according to claim 18 , wherein said estimating means estimates the position of said specific player based on the result of analyzing the game situation. 21. The image processing apparatus according to any one of claims 18 to 20 , wherein the estimating means recognizes player substitution and estimates the position of the specific player. The control means controls at least one of exposure adjustment and focus adjustment of the specific target based on the position information of the specific target acquired by the acquisition means, and the specific target acquired by the acquisition means. 22. The image processing apparatus according to any one of claims 1 to 21 , characterized in that it is possible to select a mode in which control is not based on target position information. 2. A mode according to claim 1 , wherein said control means is capable of selecting a mode in which said additional information is displayed or a mode in which said additional information is not displayed when the position of said specific target is outside the display screen of said display means. image processing device. The specific target is a specific player in a predetermined game, and the control means is capable of selecting a mode in which the additional information is displayed or a mode in which the additional information is not displayed based on the result of analysis of the game situation. The image processing device according to claim 1 . display means for displaying an image;
selection means for selecting a specific target from the screen displayed on the display means;
Designation information generation means for generating designation information regarding the specific target selected by the selection means;
transmitting means for transmitting the specified information generated by the specified information generating means to a server;
Acquisition means for acquiring from the server the location information of the specific target generated by the server based on the specified information;
Control means for controlling at least one of exposure adjustment and focus adjustment of the specific target based on the position information of the specific target acquired by the acquisition means;
tracking means for tracking the specific target after the position information of the specific target is obtained by the obtaining means;
The image processing apparatus, wherein the control means can select whether or not to operate the tracking means when the position of the specific object is outside the screen of the display means. display means for displaying an image;
selection means for selecting a specific target from the screen displayed on the display means;
Designation information generation means for generating designation information regarding the specific target selected by the selection means;
transmitting means for transmitting the specified information generated by the specified information generating means to a server;
Acquisition means for acquiring from the server the location information of the specific target generated by the server based on the specified information;
Control means for controlling at least one of exposure adjustment and focus adjustment of the specific target based on the position information of the specific target acquired by the acquisition means;
tracking means for tracking the specific target after the position information of the specific target is obtained by the obtaining means;
The image processing apparatus, wherein the specific target is a specific player in a predetermined game, and the control means can select whether or not to operate the tracking means based on the game situation. . a display step for displaying an image;
a selection step of selecting a specific object from among the images displayed by the display step;
a specific information generating step of generating specific information about the specific target selected by the selecting step;
a sending step of sending the specified information generated by the specified information generating step to a server;
an acquisition step of acquiring from the server the location information of the specific target generated by the server based on the specified information;
A control step of controlling at least one of exposure adjustment and focus adjustment of the specific target based on the position information of the specific target acquired in the acquisition step;
has
The image processing method, wherein the control step displays the position of the specific target within or outside the display screen of the display step as additional information based on the position information. a display step for displaying an image;
a selection step of selecting a specific object from among the images displayed by the display step;
a specific information generating step of generating specific information about the specific target selected by the selecting step;
a sending step of sending the specified information generated by the specified information generating step to a server;
an acquisition step of acquiring from the server the location information of the specific target generated by the server based on the specified information;
A control step of controlling at least one of exposure adjustment and focus adjustment of the specific target based on the position information of the specific target acquired in the acquisition step;
Having a tracking step of tracking the specific target after obtaining the position information of the specific target by the obtaining step;
The tracking step executes tracking of the specific target after obtaining the position information of the specific target from a server, and requests transmission of the position information to the server when tracking fails. Processing method. A computer program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 26 . 30. A computer readable storage medium storing a computer program according to claim 29.

Images