Causes of engine surging/hunting at speed?

Kawasaki FB460V engine, which has a speed governor so normally throttle set to max and the governor controls the throttle to mainatin speed.
 
Idling is fine bit at higher speeds the engine surges quite significantly - maybe plus or minus 500 rpm? 
 
Its an old engine (about 1990 I think) so i bought a new carb and its now worse, almost sounds like a kid on a motorbike revving it!
 
I discovered by accident that if I remove the air filter it almsot cures it.
 
Spark plug s normal brown colour so i think the mixture is broadly correct.
 
The idle jet adjustment screw is very difficult to get at without removing the carb. I set it to the service manual recommendation of 1-1/8 turns out from fully closed. However i notice on the old carb it was nearly two or three turns out. But surely this would have no bearing on the fast running anyway?
 
Whats really frustrating is that when i replaced the old carb it hasnt improved.
 
Ive got a suspicion that I need to back off the idle screw a lot more, but with the difficulty in trial and error I am looking for some guidance.
 
Could it also be anything else? Valves, timing etc?  I'm not the original owner but i doubt its had a proper service for years!
 
Thanks