When you're editing a media source, you should see an Access Permissions tab where you can define what user groups have access to the media source. If no permissions are set, everybody gets access. Once permissions are set, only those groups with access (and sudo users) will have access.