I went to school briefly when I was younger and, yep, found the experience to be both boring and frustrating. And yes, a waste of time. It was the prep school of a supposedly uber-academic grammar school but it was nowhere near as demanding or challenging as I expected and wanted it to be. For instance, even though I was 7 and had been reading and writing since I was 3, the class teacher still insisted on spending (?wasting) an entire morning teaching me how to tell the time - something I could already do perfectly well and had been doing for myself for as long as I could remember. That's just one example. After about a month of being there I actually stood up and told the school principal that she and her entire school were wasting my time and that if all I wanted to do was muck around all day, I might just as well have stayed at home ('home' at the time was parents' accomodation at the kids hospital rather than our home proper).
I also think that if I want to waste my own time at home, that's fine. That is my choice and I'm free to make it for myself just as long as it does not impact on anyone else. One thing I did learn at school was that they do not feel obliged to hold themselves to the same standard of courtesy and respect for anyone beyond themselves. They were perfectly content to waste my time for me!
I don't reckon there is ever going to be a perfect answer to your question though because as someone else already pointed out the idea of what is and isn't wasted time is pretty subjective. "One man's bread is another man's poison" as the saying goes. One only has to read a selection of posts here to know that what one person loves about going to school, another one hates; and what attracts some kids to school, repels others.
I think a lot must also depend upon the individual's own preferred learning style, as well as their own educational aims and objectives; and there home-ed is probably always going to have the edge over mass schooling. After all, you can't please all the people all of the time whereas pleasing one person is pretty straightforward! If what goes on in school does not suit your particular learning style nor tally with your individual aims and objectives for your education nor goals for your future then pretty much every moment of every day is going to feel like a waste of time.
Interestingly our state's own Department of Education actually publishes, in its guidelines for home education, its own acknowlegement that much of the time spent during the average school day in its own schools is just "empty time" when compared to the community based alternatives such as home education.
Finally I suppose the question ultimately comes down to just what an individual does/does not consider empty time, and whether or not that individual also considers such empty time to be wasted time.