"and it was on sovereign Confederate territory."
WRONG.
The legislature of the state of South Carolina had in, IIRC, 1819 deeded the land to the Federal government on which the FEDERAL government built Ft. Sumpter.
Why?
The port of Charleston wanted protection from possible hostile invasions.
The state of South Carolina was demanding back what no longer belonged to it.
What specious logic! If NYC leased Central Park to Imperial Japan in 1935, I guarantee they'd not permit them to maintain a military base there in 1942.