Spontaneously Alive and Well
The problem, for me, is less that the filler was there and more that they didn't use it to develop the characters outside of Garnet.Honestly I think the filler episodes is what really killed a lot of the show for me. Imagine if shit like Dragonball Z did that. We go from actual cool shit like fighting a genocidal alien overlord to suddenly watching Goku take his family to the beach, then the next episode is him going camping with Krillin, then we get to watch Goku tag along on a date with Bulma and Vegita, then we get to watch even more meaningless stupid bullshit before we ever get back to anything remotely interesting.
Yeah, I know DBZ had "filler" to pad out the episodes while the manga was written, but at least they tried to focus on the interesting shit in the story with the filler instead of wandering off on autistic tangents nobody cares about.
To me, if you're going to include filler in your series, the least you can do is reward the viewer for watching them by either having the character explain more about their past or having them build their relationship with the people around them.
If the viewer is better off skipping them, then that filler was pointless.