In swarm research, a new method makes it possible to track the eye movements of fish automatically, non-invasively and in 3D. Researchers from the Cluster of Excellence "Collective Behaviour" and the Max Planck Institute of Animal Behavior in Konstanz explain why this is important for understanding the "rules of the swarm".

Schools of fish are mesmerizing examples of collective animal behavior. Thousands of individuals move in near-perfect synchrony, despite each fish having only a limited view of their surroundings. How do they achieve such effortless coordination with minimal local information? And what visual cues does an individual fish actually perceive within the shifting, dynamic group?

Researchers from the Cluster of Excellence "Collective Behaviour" and the Max Planck Institute of Animal Behavior want to answer these questions and developed a new technology that looks deep into the fish's eye: Their new 3D eye tracking method is non-invasive and makes it possible to track the eye movements of free-swimming fish in a fully automated way, using only video recordings of the fish.

Why the fish eye matters
Why is it important to know the retinal view of the fish in 3D? Quite simply: In order to help us understand the "rules of collective behaviour", we need to estimate what information is available to the individual members of the swarm. For example: The school’s collective behaviour is based on thousands of decisions that each individual fish makes. If a fish changes its direction of movement, what factors influence this decision? Which other fish were in its field of vision that it used for its orientation? Which fish, in turn, will react to this one? What rules of behaviour do fish collectives follow, and what sensory perceptions are their decisions based on?

The scientists at the Cluster of Excellence "Collective Behaviour" investigate the nature of interactions among individuals in collectives, from fish schools to flock of birds, from locust swarms to the large herds that migrate across continents. Typical for their research setups, cameras film the animal collectives, in the lab or in the field. The researchers then analyze the recordings using computer vision: The position and body posture of each individual member of the group is tracked and evaluated by a computer algorithm every few milliseconds and compared with each other. An important aspect is reconstructing the field of vision of each individual, as this allows us to understand what each individual perceived and whether that influences its subsequent movements. And this is where the new eye tracking method comes into play.

Reconstruction of the visual field
It is not that easy to reconstruct the field of view of free-swimming fish. Just tracking the position of the eyes is not enough; the eyes must always be set in relation to the animals' body posture. For the scientists, it was especially important that no interventions on the animals were necessary, for example, that they did not have to be fitted with an eyepiece.

"Our new non-invasive method meets all these requirements", says Liang Li, who played a key role in developing the technology. "Using the camera images, we reconstruct firstly the 3D body posture of the fish, secondly the precise position of the eye - which like for humans, can move in the eye socket - and thirdly we reconstructed their retinal view, to see what they see”.

A major advantage of the new method is that the behaviour of the fish is analyzed in 3D. Previous methods were generally based on 2D images and therefore did not fully reflect the three-dimensional nature of fish schools. In addition, no intervention on the fish is required: they are simply recorded by cameras when swimming freely in the tank. At least two cameras are required, but, using many cameras, the system can be expanded to large fish tanks – the more cameras, the more precise is the analysis.

"Understanding how animals perceive their environment and interact with social partners is critical for unravelling the mechanisms underlying collective behaviour", says Liang Li. "Our method enables precise access to the visual perception of fish who move freely".

Using the new method
The new method has already been used in initial behavioural experiments with goldfish, examining the field of vision of an individual following a fellow goldfish swimming ahead. "The reconstruction of the retinal view revealed that goldfish dynamically adjust their eye movements so that the fish swimming ahead remains constantly in the centre of their retina", explains Ruiheng Wu, lead author of the corresponding publication in the journal Communications Biology.


The researchers also noticed negatively synchronized eye movements, which means that the eyes move in opposite directions rather than working together in parallel. For example, if the left eye focuses on the neighbouring fish and follows its movements, the right eye often turns in the exact opposite direction. In future experiments, Liang Li wants to find out whether this is also the case with other fish species – and whether both eyes of predatory fish ultimately align when focusing on their prey.

The eye-tracking method is described in detail in the journal Communications Biology: https://www.nature.com/articles/s42003-024-07322-y

Key facts: Original publication: Wu, R., Deussen, O., Couzin, I.D., Li, L. Non-invasive eye tracking and retinal view reconstruction in free swimming schooling fish. Commun Biol 7, 1636 (2024). https://doi.org/10.1038/s42003-024-07322-y Joint project of the Cluster of Excellence "Collective Behaviour" at the University of Konstanz and the Max Planck Institute of Animal Behavior The Centre for the Advanced Study of Collective Behaviour at the University of Konstanz is a global hotspot for the study of collective behaviour across a wide range of species and across scales of organization.