It wasn't just that some Americans did not believe in educating Black people; they did not think Black Americans COULD be educated.
This is back when slavery was still in place in half the country. New York City was home to the African Free School system, and two boys from that system grew up to push for far greater educational opportunities.
Henry Highland Garnet and James McUne Smith are the headliners in a book by Anna Mae Duane, Educated for Freedom: The Incredible Story of Two Fugitive Schoolboys Who Grew Up to Change a Nation.
Their challenge to educational hierarchy and orthodoxy, and to each other's ideas, is discussed in this interview from 2020.