How would would you deal with slavery and Jim crow under this model?The problem isn't just size, it's also centralisation. One of the concepts the framers of teh constitution were trying to implement was the idea of localism, where most government happens as close to the governed as possible. It's why the fed was deliberately crippled its powers enumerated in such a limited way. The federal government was meant to be weak. Most day to day government was supposed to take place at the town, city, county or state level, with the federal government only acting over interstate matters and on external matters that affected the entire union.
The reason the federal government has bloated up so hugely today is because power has been drawn to the centre. Reform of the government would have to start by moving that power back to the fringes, where it belongs, which would obviate the need for enormous, centralised bureaucracies entirely.
It'll probably never happen, but it should.