Exactly, if I now see a tweet in my Twitter timeline with an Instagram link, instead of what used to show the actual photo, then I simply don't usually bother clicking the link. All this has achieved is, like you say, reduced traffic to their site.
Instagram's reasoning was that they wanted people to visit their site to view the photos, but it's probably having the opposite effect as it's one extra click that most people can't be arsed to do just to see someone's lunch or photo of their cat/dog/hamster.
By all means they can have a private disagreement between themselves and Twitter, but that's just cutting their own nose off to spite their face!
