Long-Term Tracking

By Žan Pušenjak

Advanced Computer Vision Methods, FRI, University of Ljubljana

May 20, 2025
computer-vision object-tracking deep-learning

Short-term tracker SiamFC

We were tasked with improving the pretrained SiamFC(Bertinetto et al. 2021) short term tracker to give it long-term tracking capabilities.

We compared the results of both implementations and commented on the changes. We used Precision, Recall and F-score metrics.

Long-term tracker extension

We achieved the long-term functionality by thresholding the SiamFC maximum response.

Set threshold $threshold \gets \theta$ Get maximum response $max\_res \gets network(candidates)$ Start detecting Track repeat

So if the response would fall under the threshold while tracking the tracker would know that target has left the frame and start detecting by sampling over the entire image[lt-track]. When detecting if the response of the random samples would be sufficiently high, the tracker would know the target has reappeared and would start to track the target again[lt-detect].

Example of the tracker running (with improvements) can be observed on 3 and 1

Double threshold

The addition of the threshold had the problem. If the threshold was too high the re-detection was working, but the tracker would lose the target too often, whereas if the threshold was too low, the tracker would correctly detect target leaving the frame, but would too quickly re-detect wrong object as the target re-entering the frame.

We fixed this issue by introducing a double threshold approach. When tracking the threshold for losing the target would be lower, and when the target would leave the frame the threshold for re-detection would be much higher in order to only start tracking when the detection was confident.

Since different sequences require different thresholds, we set the thresholds based on the first frame maximum response $\theta_0 = \lambda_0max\_res$ for target leaving the frame and $\theta_1 = \lambda_1max\_res$ for re-detection threshold.

and than dynamically updated them based on current best response based on $\alpha$ parameters according to formula $\theta_i \gets (1-\alpha)\theta_i + \alpha\lambda_i max\_res$ Where the appropriate $\theta$ was used if the tracker was detecting or tracking.

The parameter values we ended up were $\theta_0 = 0.4$ , $\theta_1 = 0.66$ and $\alpha=0.0001$ .

Example of re-detection of `person14` sequence.

Sampling

When the target is detected to leave the screen the tracker starts sampling the image for target re-appearance. The number of samples depends on the target size. So if the target size is bigger there will be less samples than if the target is small. The number is calculated as such $N = \frac {hw}{h_tw_t2}$ where $h$ and $w$ are the hight and width of the entire frame and $h_t$ and $w_t$ are target hight and width. The number is capped at 250.

We saw that sometimes the tracker would fail and get stuck on a very small or big target size and thus was not able to recover. We tried to fix that by additionally sampling the scale of the sample detections. The scale change was dependent on the number of samples (and thus to the size of the target). So if the target was extremely small we would allow for bigger scale increase and limit the decrease and if the target was big we would allow for an bigger scale decrease and limit the increase[sample_scaling].

Get number of samples $N$ $scale\_up \gets2.5$ $scale\_down \gets1$ $scale\_up \gets1.5$ $scale\_down \gets0.5$ $scale\_up \gets1$ $scale\_down \gets0.25$ $scale_i \gets sample(from=scale\_down,to=scale\_up)$

The working of the sample scaling can be observed on Figure3 or Figure1.

Gauss sampling when target leaves frame on one side.

First we implemented uniform sampling and later added gauss sampling with an increasing standard deviation. The standard deviation increased each frame of the re-detection, but was capped at $75\%$ of width or hight. The problem with gauss sampling is, that it samples around the last confident center, so if the target leaves the frame by going to either side, most of the samples will remain on that side. Like on Figure4, where the target left the frame on the left side. If the target re-appears on another side (example frame cut in sequence sitcom this will pose a problem).

Example of re-detection of `car9` sequence. Using uniform sampling

Example of using increasing gauss sampling on `car9` sequence.

Results

On Figure5 we can see the double threshold working when the target gets occluded (The actual detections can be seen on Figure3.

A more diverse threshold graph for sequence cat1 can be seen on Figure6. We can see that using only one threshold would be difficult, since the re-detections or target leaving the frame would be very hard to get right.

Redetection threshold changes on sequence `car9`

Redetection threshold changes on sequence `cat1`

The final evaluations can be observed in Table1. We can see that long-term tracker has much higher recall. The drop in precision happens, because when the tracker detects that the target has left he frame, it starts reporting $[0,0,0,0]$ for the bounding box. Because the short-term tracker still reports the most likely location it probably gets some intersection over union unwillingly which boosts its score. An example of this can be seen on Figure3 on the third frame, we can see that the tracker gets the best response form the car (blue bounding box), but the response is not strong enough yet, so it still reports $[0,0,0,0]$ . If I would report the most likely position, those frames would contribute to the precision.

Results of different trackers
Tracker	Precision	Recall	F-score
short-term	0.62	0.31	0.42
long-term (uniform sampling)	0.60	0.41	0.50
long-term (gauss sampling)	0.54	0.46	0.49

Additionally we can observe that gauss sampling slightly worse overall but achieved a better recall.

Bertinetto, Luca, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H. S. Torr. 2021. Fully-Convolutional Siamese Networks for Object Tracking. https://arxiv.org/abs/1606.09549.