Habitat loss and fragmentation can lead to smaller and more isolated populations and reduce genetic diversity and evolutionary potential. Conservation programs can benefit from including monitoring of genetic factors in fragmented populations to help inform restoration and management. We assessed genetic diversity and structure among four major populations of the Cactus Wren (Campylorhynchus brunneicapillus) in San Diego County in 2011–2012 and again in 2017–2019, using 22 microsatellite loci. We found a significant decline in heterozygosity in one population (San Pasqual) and a decline in allelic richness and effective population size in another (Sweetwater). Genetic diversity in the remaining two populations was not significantly different over time. Local diversity declined despite evidence of dispersal among some populations. Approximately 12% of genetically determined family groups (parents, offspring, siblings) included one or more members sampled in different territories with distances ranging from 0.2 to 10 km. All but one inferred dispersal events occurred within the same genetic population. Population structure remained relatively stable, although genetic differentiation tended to increase in the later sampling period. Simulations suggest that at currently estimated effective sizes, populations of Cactus Wrens will continue to lose genetic diversity for many generations, even if gene flow among them is enhanced. However, the rate of loss of heterozygosity could be reduced with increased gene flow. Habitat restoration may help bolster local population sizes and allelic richness over the long term, whereas translocation efforts from source populations outside of San Diego may be needed to restore genetic diversity in the short term.