For several species, refuges (such as burrows, dens, roosts, nests) are an essential resource for protection from predators and extreme environmental conditions. Refuges also serve as focal sites for social interactions, including mating, courtship, and aggression. Knowledge of refuge use patterns can therefore provide information about social structure, mating, and foraging success, as well as the robustness and health of wildlife populations, especially for species considered to be relatively solitary. In this study, we construct networks of burrow use to infer social associations in a threatened wildlife species typically considered solitary—the desert tortoise. We show that tortoise social networks are significantly different than null networks of random associations, and have moderate spatial constraints. We next use statistical models to identify major mechanisms behind individual-level variation in tortoise burrow use, popularity of burrows in desert tortoise habitat, and test for stressor-driven changes in refuge use patterns. We show that seasonal variation has a strong impact on tortoise burrow switching behavior. On the other hand, burrow age and topographical condition influence the number of tortoises visiting a burrow in desert tortoise habitat. Of three major population stressors affecting this species (translocation, drought, disease), translocation alters tortoise burrow switching behavior, with translocated animals visiting fewer unique burrows than residents. In a species that is not social, our study highlights the importance of leveraging refuge use behavior to study the presence of and mechanisms behind non-random social structure and individual-level variation. Our analysis of the impact of stressors on refuge-based social structure further emphasizes the potential of this method to detect environmental or anthropogenic disturbances.