The following quote from Carl Hopkins does a nice job of describing the modern concept of neuroethology. The quote is borrowed from the course description page of Carl's neuroethology course at Cornell. You should also check out the UIUC neuroethology course that I've taught over the years. The model systems page has descriptions of the neural bases of behavior for about 20 different animals.
What is neuroethology? [ by Carl Hopkins]
A good question. Many have tried to define and describe it. Here is my personal take on the subject.
Neuroethology is the biological approach to the study of the neural basis of behavior. Thus, the focus is on the role of the nervous system in behavior, but the perspective is that which is called 'ethological'. The ethological approach emphasizes the causation, the development, the evolution, and the function of behavior and neuroethologists seek to understand this in terms of neural circuits. Neuroethology is the study of natural behavior, which, in the older scientific literature, was called "instinctive behavior" or "innate behavior". Neuroethologists base their studies on behavioral studies that often are done in the field on the animal's own turf.
The neural approaches used in neuroethology are as diverse as the field of neuroscience itself. Thus, some neuroethologists use behavioral methods only to ask profound questions about the organization of underlying neural circuits. Some use intracellular recording techniques to probe one cell at a time in some neural circuit that is involved in a particular behavior. Some use neuroanatomical techniques, some use comparative methods to look at how nervous systems differ from one species to another. Molecular methods are used to explore phylogenetic relationships. All sorts of molecular and cellular methods are used to explore neural connections and to uncover diversity and specialization of nerve cells. Computer modeling is used to predict behavior of neural circuits.
The subjects in neuroethology are as diverse as the animal kingdom. Neuroethologists are interested in comparative aspects of behavior and in the evolution of the nervous system. The comparative method is used in many studies. A typical assemblage of neuroethologists is likely to talk about jellyfish and corals, sea squirts and sea slugs, insects and insectivores, slugs and bugs, birds and bats, frogs or toads, and even mammals. Some may be interested in behavioral processes, not a particular group, others in neural computation and algorithms, not any particular system. Somehow the field holds together by the mutual attraction of neuroethologists to like-minded concern for diversity of animal nervous systems and their role in behavior.
Want to learn more?
Zupanc GKH (2004) Behavioral Neurobiology: an integrative approach. Oxford Press.
An excellent, up-to-date undergraduate textbook that introduces the field through chapters devoted to the history and tools of neuroethology, spatial orientation and bat echolocation, motor control of swimming in tadpoles, prey and predator recognition in the toad, sensorimotor integration in weakly electric fish, neuromodulation, migration and homing, communication in crickets, and cellular mechanisms of learning and memory.
Carew T (2000) Behavioral Neurobiology: The Cellular Organization of Natural Behavior. Sinauer Associates.
Another excellent undergraduate text that introduces the field through chapters devoted to specific model systems, including bat echolocation, prey localization in barn owls, visual feature analysis in toads, mate calling in crickets, locust flight, crayfish escape, songbird learning, honeybee foraging, reflex conditioning in sea slugs, learning and memory in fruit flies, and spatial navigation in rats.
Camhi J (1984) Neuroethology: Nerve Cells and the Natural Behavior of Animals. Sinauer Associates.
This classic introductory text is currently out-of-print, although you may be able to find copies at your local library or through an online book service.