I think that would be a good move, or at least something to try. And the good news is you can always test the before/after with Analytics. My blog used to be made up of all categories on the nav, but I have slowly moved to more static menu links because of the very reasons you state. Plus Analytics said no one was clicking on most of the categories.
I'm actually in the process of making all the menu options on my blog static/landing pages with tiers. It just makes more sense. After all, you can always link to your posts in that category on the landing page that you link to. There are plugins for that.
I think what determines how you should structure your site depends on what your visitors need. For example, I've sort of branded my blog as my place to talk about the latest and greatest so my "What's New" tab gets a LOT more clicks. People are sort of trained to start there. I left the "Make Money" and "Traffic" category tabs there because they were the most-clicked categories. But I still am going to turn those into static/landing pages with better "getting started" content to draw people to the appropriate posts in that category.
So ask yourself, what is the most important content on your site? How can you structure your menu so your visitors are finding the most important information? I used to be big on creating numerous categories on my blog, but over the years found out it made more sense to limit the number of categories and work more on the layout.
Remember, it's about what your visitors want. Start there and let that dictate what you should do with your nav. I don't think there is one right answer here.
Here's the other thing to consider. When you are looking at your bounce rate, make sure you look at each page individually instead of the site average. For example, my hex color page gets a lot of traffic but the bounce rate is always high. Some pages will have high bounce rates by nature because people find what they need and leave. Poople just want to generate a color and then they're off.
But since that page gets a lot of traffic it drives my entire site's bounce rate down. So my homepage bounce rate may only be 50% but a high-traffic page with a 85% bounce rate can drive the entire bounce rate down. This is why Google says don't worry much about the average bounce rate with regards to SEO. It's definitely RELATIVE to the search term. Now that doesn't mean bounce rate isn't important. It tells a lot about how helpful your site is to people! For example if your homepage bounce rate is 85% that is alarming. But certain searches yield higher bounce rates due to the nature of the search. So that's why you need to look at bounce rate on an individual basis.
Great question, though! You should have enough for a site review now.