In this paper, we present a method of automatic video surveillance using a thermal camera for protection of critical coastal areas and port facilities. Out method uses particle filter that has likelihood function obtained by eigen space method. Furthermore, multiple sets of particle are introduced and each set is used to detect and track separately one object. To eliminate duplicated tracking , our method deletes the redundant set by comparing two directions that indicate the each direction of largest possible variance obtained by Principal component analysis. Finally, some experimental results are shown to evaluate the validity of our method.