Yes they can - especially if a lot of commenters have been approved and are loading on popular individual posts/pages.
This article details a number of easy and fast workarounds to Gravatar-related issues:
http://shibashake.com/wordpress-theme/gravatars-can-slow-down-your-wordpress-blog