And site speed is a part of the mobile responsive experience - site speed even has its own category in Analytics. Mobile is a market trend that seems indisputable, I know some people resist the hassle of converting their sites but it is worth it, and not actually very difficult.
I would also add that image size is something we need to take into account for mobile device sites. I have to remember to think about the limited processing power of those devices and make sure when I add images I scale the images before uploading, to keep the file size to a minimum. A large image with small HTML dimensions is still a large image, and this taxes the load time on mobiles.
Think what a poor user experience it is when you have to load up a non responsive site with lots of images. I think all sites will becomer meaner and leaner in their design over the next few years, and code will be minified more like Twitter bootstrap. I think the graphic layout of sites are going to go through a Renaissance with the advent of CSS3. I have seen incredible sites built with CSS3. Being able to do what you can do with CSS3 could spell the end of tiled background gifs and teselatin patterns. That alone would make the site faster.