You've got some good points here, as it does get hard to follow at times. I remember seeing it for the first time and being lost around the middle, wondering if it even had a point. Visually, it's find it a very striking film, apparently it's one of the few Gilliam films in 2.35 aspect ratio...
So while I agree that it does meander, I wouldn't necessarily say it's unwatchable. From what I understand, Gilliam's more recent films tend to lose focus even more.
Speaking of which, Fear and Loathing is the only Gilliam film I've seen. I remember seeing Time Bandits when I was younger, but probably too young so I didn't quite get it. What are some stronger Gilliam films that you would recommend?