I guess it depends on what the concern is.
If the only concern is a theoretical ultimate bioweapon that wipes out a population then I guess you've got a point.
If the concern is someone cooking up something bad at home (which for some reason I took to be the thrust of the thread) then I don't think it has to be the absolute most potent or infectious thing in the world, it just has to be nasty enough to matter. Plain old smallpox would probably qualify, even if a response could be mounted pretty quickly.
COVID has done enormous damage to the world with R0 estimated well below 10 and an extremely low IFR. Granted, much of the damage was done by governments in response to the perceived threat, but that's true of most terrorism.
I don't think it would be hard to culture anthrax or botulinum at home, and probably several other toxin-producing bacteria. And you can grow ricin or abrin in your garden. (abrin is related to ricin but much more potent) Weaponizing it is the hard part.
