Not everyone I know has liked the film as much as I did. Some say that it's too slow and takes its time (when I felt that it was a way to establish the lyrical and understated tone of the story), others have complained that the soundtrack sometimes overwhelms the dialogue (I'm not sure I noticed this though Heath Ledger's character, in particular, is so inarticulate at points that he mumbles his responses out, which might take a couple of viewings to understand). The songs chosen for certain scenes ("King of the Road!...") also capture the passage of time perfectly, even if some of the make-up used in later years doesn't quite work that well.
But it's those small details, those particular small moments, that sometimes grab a hold of you and stick with you long after watching the film, particularly if they mirror some of your own experiences. Some friends who weren't as thrilled about the movie when they saw it have told me that they've been surprised by how it grows on you after you leave the theatre.
In any case, for me, that "whoa" moment comes in during the later scenes, when Ennis and Jack are saying goodbye after one of the "fishing" trips that they schedule once or twice a year to spend time with each other. Jack makes no qualms about his desire to leave his wife and kids, if only Ennis will join him in moving together to the ranch owned by Jack's parents. But he is also frustrated by Ennis' reluctance to make the move - to grab life by the balls - by moving in with him, and tells Ennis that he can't go on just seeing each other every 4 years. Ennis, who says he has to stand by the responsibilities of adult life, the choices he's made, simply can't. I wish I could quote that "every 4 years" speech line by line because some things in it struck close to home from past experience. For others it might be something else.
In any case, when I posted "Back to Brokeback" a few days ago I said I was awaiting for CP to review the film. Well, what stuck with him was... the wrestling. More about it here (thanks, CP, as well, for the plug - and for your amazing blog page).
UPDATE: J. Bernard Jones has written a comprehensive review of the film on his The Edge of Night blog, for those who might be interested in reading a little more about the film.