I voted for the movie to stay, but it was a closer call than I expected going in. This movie really is a tough sit, a lot of time spent with very unpleasant characters in a not particularly propulsive narrative. The only reason it works at all is probably the superlative craft at play: the direction, acting, photography, editing, sound design, etc. And I would also argue that the film clearly gives the impression that the filmmakers know it's about unpleasant people and want that to be the point you take away from it, as a way of subverting the usual heroic boxing movie narrative about an underdog coming back to win. Nope, this time the guy reaches the top fairly quickly, then slowly deteriorates because of his own psychological issues. This is usually what a lifetime of punching and getting punched for a living actually results in.
I do agree with Paul & Amy that #4 seems too high a placement. Scorsese's other AFI 100 movies carry a lot more cultural cache and influence than this one IMO, as well-made as it is. I'd bump this one further down.