Not very familiar with the 2004 but have you tried cleaning the throttle body? Sounds like something might be sticking.

 

While checking the throttle body, look at the linkage which makes the throttle body move. Without starting your V, open the hood & have someone inside step on the accelerator pedal. Visually check to see if there is some binding on the throttle body linkage. Then do the same with the vehicle engine running to notice any difference in the linkage action.

Just might need a spray of brake cleaner to remove some crud on the outside linkage that makes your throttle body "sticky".

 

Just an update in case it helps anyone else.
I removed the airbox and ducting and cleaned the throttle body. Used carb cleaner and a cloth, was very black and sticky inside.
No problems found with linkage or ducting.
Did idle setup as battery was disconnected.
Idle now slightly lower and smooth and original problem of high idle with foot off pedal much reduced.
Thanks for previous replies, will leave it at that as much better.

 

Just an update in case it helps anyone else.
I removed the airbox and ducting and cleaned the throttle body. Used carb cleaner and a cloth, was very black and sticky inside.
No problems found with linkage or ducting.
Did idle setup as battery was disconnected.
Idle now slightly lower and smooth and original problem of high idle with foot off pedal much reduced.
Thanks for previous reports, will leave it at that as much better.

Glad you have improved it! Thanks for letting us know, this might help another CR-V owner.