The impacts of global climate change on agricultural production and food security are serious source of worry to farmers in sub-Saharan Africa. This is because their economies mainly depend on agriculture which is now affected by climate change catastrophes. The paper examines why farmers find it difficult to adapt to climate change impacts. A multi-stage random sampling technique was used in selecting 120 farmers that participated in the study. Both descriptive and inferential statistics were used in analyzing data. Major results show that the most difficult challenges faced by farmers in adapting to climate change impacts in the State were: lack of improved agricultural technologies, low adaptive capacities, and unsustainable agricultural practices. Furthermore, a framework for climate change adaptation shows that a mix of agronomic best practices, technology and innovation development and institutional and policy reforms were proposed as capable of improving farmers’ adaptation capacity to climate change. The paper concluded with the recommendation that farmers’ adaptation capacities to climate change need to be urgently strengthened by extension service so as to sustain agricultural production and food security even in the event of climate change.