Your 'downtime' scenes have to advance the story just as much as a chase or confrontation does. They advance the intellectual/emotional aspects. Never think of them as filler. Give them purpose.