It's not the hands...

It's the talking.

(1) Cell phone use damages driving performance almost to the DUI level.
(2) Laws against using cell phones while driving are effective and reduce cell phone use a lot.
(3) Where this happens, there’s no reduction in accidents!
Toward the end of the story we get what I think is the key to the puzzle: hands-free cell phone gadgets, which are legal (but shouldn’t be) everywhere, because talking on the phone with a hands-free device is just as bad as holding the phone to your ear.  The problem with a cell phone in the car is not dialing and fumbling, which after all take only a small fraction of the time required for a phone conversation (and you can do that at a time of your choice), but mental distraction of a particular kind.   This is surprising because listening to the radio, or chatting with a passenger, are not dangerous; why should chatting with someone on the phone be so risky?
To understand why, consider a couple of other things that are dangerous in cars: very young children and loose pets, which both (i) require your attention unpredictably (ii) with no sense of what’s happening on the road ahead that might be more important, and (iii) you know they might do this at any time.  The radio, in contrast, doesn’t have that psychological grip; you might miss a pearl on All Things Considered if you stop listening for a minute, but you know the radio will not be insulted, nor raise its voice insistently.

The party on the other end of the phone conversation is an adult to whom you psychologically owe attention, but unlike the adult passenger, has no idea of what you are seeing through the windshield.  A passenger will subconsciously stop talking if something untoward or just complicated is unfolding on the road ahead, and will expect you to suspend the conversation similarly, so she causes no important distraction at the critical moments when you need to be driving on all neurons, and you are aware of all this. In contrast, the person on the phone can’t do either of these things, and you are aware of that as well.  When you need to navigate a tricky bit of road, there’s no time to ask someone to be quiet, and telling a peer to shut up for a minute, in any terms, is so rude that it absolutely requires an excuse that makes it take even longer (“can you hold on for a minute? one of the kids is playing with my blunderbuss and I think it’s loaded”). [More]

I wonder if farmers are more likely to engage in this behavior due to 1) low-traffic rural roads and 2) auto-steer acclimation. Regardless, it looks like a really bad habit, and one reason some companies like Pioneer (that I know of) strictly prohibit it.

[Update: Great additional info from a commenter below]