''The days of iron bars and concrete floors are gone,'' said Dr. Scott R. Derrickson, curator of birds at the National Zoo's research center in Front Royal, Va. Reassessing Behavior
The new studies of animals in natural social groups, rather than in isolation, have yielded a vast array of discoveries, prompting many psychologists to reassess the behavioral capabilities of even the most ordinary animals.
''We've learned that maybe we didn't give animals enough of a chance,'' said Dr. Meredith West, an animal behavioral psychologist at the University of North Carolina and an expert on the song patterns of cowbirds. ''Now we're trying to find the right conditions to elicit capacities in animals we didn't know they had.''
Naturalistic research enclosures have been around for years in universities and at zoos in Washington, San Diego and the Bronx.
But zoos and their huge collection of animals are increasingly teaming up with university research staffs to conduct joint studies, or to develop research facilities of their own.
Some changes in habitat design and conditions are the result of aggressive lobbying by animal rights groups, but researchers say that improved physical and social conditions arise also from the quest for greater insights into animal behavior.
The Audubon Zoo in New Orleans, for example, has increased its emphasis on research and seven months ago hired its first research director. The zoo is planning to use its five-acre swamp exhibit to study how sunlight affects white alligators and how the animals and their naturally colored brethren develop differently.
Neither study could have been conducted in the wild, said Dr. Edward F. Gibbons, Jr., the zoo's new research director. ''There is more and more research at zoos because of the critical need to preserve animals in captivity, so they can eventually be returned to the wild,'' Dr. Gibbons said. At Home With the Monkeys
Zoologists, veterinarians and animal behavior psychologists who met recently at a conference organized by the State University of New York at Stony Brook said that as many as half of all studies of the behavior of captive animals now use more naturalistic habitats.
Primate cages, for example, are now often large enough to hold a whole family comfortably. In the enclosures, monkeys clamber along branches, ropes that simulate vines and Fiberglas tree trunks.
''The more we know about animals, the better we can keep them,'' said Dr. Benjamin B. Beck, general curator of the National Zoo in Washington. ''We can improve their health, reproduction capabililties and longevity.''
Field investigators use observations from the man-made settings and apply them to their research in jungles, deserts and other native habitats. Other researchers study conditions such as overcrowding that may be difficult to find in the wild.
Experts cite many examples of benefits, both for the animals and the researchers, of the new settings. Gorillas, for example, became less aggressive and groomed themselves more often. Some female cranes craved calcium shortly before laying eggs, to improve the hardness of the shell. Researchers fed them crushed oyster shells with good results.
In experiments with cotton-topped tamarins, a small monkey native to the forests of northern Colombia, Dr. Charles T. Snowdon, an animal behavior psychologist at the University of Wisconsin, has housed a family of the monkeys in a cage 9 feet wide, 6 feet deep and 6 feet high, complete with tree trunks and branches of varying size and flexibility. Earlier cages were barren cells big enough for only one animal, he said.
Placing monkeys together in the more naturalistic habitat has improved their breeding success and parenting skills, Dr. Snowdon said.
''Ideally, we'll try to release them to the forest someday,'' he said.
But researchers who reintroduce endangered species to the wild have discovered that even the most innovative enclosures are sometimes inadequate. To teach animals how to find food or negotiate slippery tree branches, scientists are giving them a kind of zoological on-the-job training.
When studying the golden lion tamarins, primates native to eastern rain forests of Brazil, Dr. Beck of the National Zoo discovered that the bedroom-sized cage with vines, branches and other vegetation was not big enough.
The monkeys did not have to work hard enough to find food hidden in the cage, and quickly developed habitual travel routes rather than learning to traverse more varied terrains.
''We needed the real thing,'' Dr. Beck said. So in 1986, six of the monkeys were allowed to roam the zoo's grounds under close supervision. And last September, Dr. Beck and his staff took 21 monkeys to Brazil.
He found that the animals had been living in a ''psychological cage.'' Born in captivity, they had always had their basic needs nearby, and thus imposed artificial boundaries on themselves when freed from their cages.
Of the 21 tamarins released into the wild last September, Dr. Beck said that 19 were still alive and were being tracked through radio collars. Why the Cowbird Sings
Psychologists say they can learn subtle details about animals in their natural social groups.
Dr. West, for example, said male cowbirds inherit a potent mating song, but rarely sing it among other males because larger birds, jealous of more melodious songs, would attack the better crooners.
Only after males were housed alone with female cowbirds, who cannot sing, did Dr. West and her husband, Dr. Andrew King of Duke University, discover the male's hidden gift. When singing his repertoire, a male cowbird would elicit a split-second wing stroke of approval from the female when he sang his best song.
But when a male was placed back in a group of males, he would stop singing his tune to protect himself, Dr. West said.
''In this case, it was just as important to study the listener,'' Dr. West said.
In earlier studies of starlings, Dr. West proved the same point. She gave baby starlings to two groups of families. She told one group to treat its chick as a human child, speaking to it the same way they would speak to an infant. She told the other group to take care of the bird like any other pet.
At the end of a year, the starlings raised like babies could repeat words and phrases, like a mynah bird, while the other group made only bird noises.