How many user in manager at same time

Hi Guys,
If I have 7 editors on in the manager at the same time the manager just grinds to a halt and the website freezes. I use Digital Ocean with 4GB Ram and 2 CPU cores. Should the manager be causing all this? All my pages are cached and run with about 1-2 queries?
Thanks

Can you tell us more about how you have the server configured?

Cloudlinux with WHM/cPanel latest.
mpm event enabled. Apache 2.2, php 5.4
pretty much everthjng set up like the phpinfo on modxcloud where I also host.

. .some resources will open and some won't and not due to locked permission scenarios.

Is CSS/JS compression turned on in the Manager? That stuff is compiled on the fly and can kill the manager due to request limits getting hit on the server in CloudLinux.

I encountered this with a web hosting company who found the following: "CloudLinux introduced a new concept of virtual memory and physical memory. The stats are showing you were nowhere near hitting the physical memory limit but the virtual memory limit gets inherited from the Apache memory usage as well apparently. We have up'd the virtual memory limit from 2GB to 8GB since it shouldn't really matter since we have a 1GB physical memory still in place. Our preliminary testing shows this seems to fix the problem."

This change completely resolved their issues. Not sure if this is directly related or not.

I've always found the biggest speed killer is cache rebuilding. When multiple editors are working in the manager and one or more of them is a click-the-save-button-every-10-seconds kind of person and I don't have something like CacheMaster installed to protect my cache from every save it will absolutely kill the site, manager and all.

I'm gonna try Jay's solution as well, I never made that connection between CSS/JS compression and manager performance.