My understanding is that any place that they have constructed, such as a parking area, you will need the pass. These are normally found at trailheads, campgrounds, etc. CNF here in Orange county does have signs stating the need for adventure pass for these places. It really depends on where you park, not where you take a dump.
If you just pull off the road-you should be fine without a pass.
I always get a yearly pass. It is my way of contributing and I think of it as a donation for a good cause. Same way I donate to DFW by getting tags I don't fill/use, I do the same with my local NF. Also add on NWTF to that list.