Usually it's because the idle air control valve (IACV) gets clogged with carbon. It either needs to be cleaned (not sprayed) or replaced.Can also be the throttle body or EGR valve needs cleaning/replacement.This is a really common problem with the ES300, see links below.
us.lexusownersclub.com