I typically build "heavy sites" though you wouldn't know it when you visit them.
The original posters question is broad and has many possible combinations of answers.
Hardware, memory, bandwidth, networks, router issues, ISP throttling, etc., all all examples of hardware related issues.
As was already mentioned, graphics, scripts, and other page content can be a culprit. Also linking from third-party sites such as Youtube and others can slow down your page, unless you implement localized solutions such as placeholder images for their content.
My book discusses how to implement minify for css and JavaScript, and I recently wrote an article on how to implement cloudflare on a MODX Revolution site (even if it is https):
http://www.shawnwilkerson.com/modx-revolution/2013/03/30/using-cloudflare-cdn-with-modx-revolution-and-still-have-ssl-pages/
I took over a project which was loading in 22 seconds on a hybrid of open source projects. I deleted the sites (there was no point to keep them) and rebuilt the sites under MODX Revolution.
MODX render time: 0.2026 s
First View 2.349s 0.596s 1.647s 1850 295 2.349s 18 863 KB 2.418s 19 863 KB
Repeat View 0.634s 0.438s 0.707s 728 295 0.634s 2 12 KB 0.938s 5 13 KB
These are without a cloud handling the static assets as we had to turn it off for the build and test stage.
I hope these give you some ideas on where to start. Web Site Optimization is an art form in and of itself and it is a huge topic. Provide us with a URL or something to allow us to give you better insight.