I suppose it would, but unless it’s a big chunk of JS I doubt it would make much difference to the user.
Also bear in mind about when you need to access it it’s functions.
The most gain you’ll probably see is in combining all the js into one include.
http://aciddrop.com/php-speedy/ has some info on this.
As long as your not doing anything invloved in you js it should not slow the page loading time.
Get the dev toolbar in firefox and turn off js and load the page, that was you can see if the js is causing a lag....