I am beginning a project to extract a couple of fields of data from someone else's website, which I will use as the basis of a calculation and then display the results on my website.
It seemed to me that by using cURL I could copy the page and write it to a file, and then parse the file to find the data fields I need.
The first step: copy the page and write it to a file.
Here's the code:
$curl_handle = curl_init();
// Where should we get the data?
$fp=fopen("assets/files/solarpage.txt", "w");
//Where are we going to store it?
curl_setopt ($curl_handle, CURLOPT_URL, 'http://www.hkelectric.com/web/AboutUs/SolarPowerSystem/RealTimeOperation/Index_en');
// This says not to dump it directly to the output stream, but instead
// have it return as a string.
curl_setopt ($curl_handle, CURLOPT_RETURNTRANSFER, 1);
// the following is optional, but you should consider setting it
// anyway. It prevents your page from hanging if the remote site is
// down.
curl_setopt ($curl_handle, CURLOPT_CONNECTTIMEOUT, 1);
// Now, YOU make the call.
$buffer = curl_exec($curl_handle);
// And tell it to shut down (when youre done. You can always make more
// calls if you want.)
curl_close($curl_handle);
// This is where id probably do some extra checks on what i just got.
// Paranoia pays dividends.
print $buffer;
fclose($fp);
//close the file and save it
I call the snippet, and the result is this: (page source excerpt):
<p>Test</p>
<p> </p>
<p><html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href='/web/SystemBusy.htm?aspxerrorpath=/web/templates/solarpower/SolarPowerDisplay.aspx'>here</a>.</h2>
</body></html>
</p>
<p> </p>
<p>Tested</p>
and I get a file created, but with blank content.. Of course the link is to something non-existent on my site.
The page I am looking at isn't "busy" when I visit it with a browser.
The only similar report I can find on the web is in Russian, and the translation makes no sense, so I'm a bit stuck. It may be of course that there's something off happening on the source website, but before I start hunting that idea down: has anyone seen this kind of thing before?
Grateful for any pointers
Nic Boyde
Evo 1.05
PHP 5.3.6
MySQL 5.1.56
