Memory and iterating through getCollection

45 Posts

Beniamin Reply #1, 13 years, 3 months ago

Hello all,

Here's my setup: I've created a dynamic XML feed using xPDO getCollection and iterating through the array, outputting each item. If no query string is passed to it, it is supposed to return the entire collection of 545 items. You can also pass a date in Unix time to the file, and it will give you everything that has been edited since that date.

It works great, except when I try to get the whole collection. At first I was getting the "Allowed memory size...exhausted" error. I increased memory_limit from 32 M to 64 M. Now it will do about 90 items and then quit, giving me an incomplete XML file.

At first I was using a foreach loop, but then I switched to a for loop to try to save memory, but it doesn't seem to be making much difference. Is my script still running out of memory? Something's going on, because it's not just the loop breaking after about 90 items, the whole script stops.

If anyone has any suggestions or guidance, I'd greatly appreciate it.

Thanks!

5,888 Posts

Bruno17 Reply #2, 13 years, 3 months ago

whats your code?

-------------------------------

you can buy me a beer, if you like MIGX

http://webcmsolutions.de/migx.html

Thanks!

45 Posts

Beniamin Reply #3, 13 years, 3 months ago

Here's the code. I would try cutting it down to just the essential parts, but I have no idea if some minor detail might be eating up memory, so this is the whole thing.

$dateLimit = $_GET['datelimit'];

$c = $modx->newQuery('modResource');

if($dateLimit != null) {
	$c->where(array(
		'parent:IN' => array(1387,15,110,220),
		'editedon:>' => $dateLimit
	));
} else {
	$c->where(array(
		'parent:IN' => array(1387,15,110,220),
		'editedon:>' => 0
	));
}
$c->sortby('editedon','DESC');
$resources = $modx->getCollection('modResource',$c);

echo '<?xml version="1.0" encoding="UTF-8"?>';
echo '<items count="' . count($resources) . '">';

 
$keys = array_keys($resources); 

for($i=0, $n=count($keys); $i<$n; ++$i) {

	$resource = &$resources[$keys[$i]];

	$parent = $resource->get('parent');
	$type = '';
	$content = '';
	$itemDescription = '';
	
	$properties = $resource->toArray();
	
	if($parent == 1387) {
		$type = 'video';
		$itemContent = $resource->getTVValue('youTubeId');
		$itemDescription = $resource->get('content');
		
	} elseif($parent == 220) {
		$type = 'audio';
		$itemContent = $resource->getTVValue('audioFilename');
		$itemDescription = $resource->get('content');
		
	} else {
		$type = 'text';
		$itemContent = "<![CDATA[" . $resource->get('content') . "]]>";
		$itemDescription = $resource->get('description');
	}
	
	$tagInput = $resource->getTVValue('tags');
	
	// Tag processing
	
	if ($tagInput == '') { return ''; }
	$tagOutput = '<tags>';
	$tags = explode(',', $tagInput);
	foreach ($tags as $key => $value) {
		$tagOutput .= '<tag>' . $value . '</tag>';
	}
	$tagOutput .= '</tags>';
	$properties['tags'] = $tagOutput;
	
	// End tag processing
	
	$properties['type'] = $type;
	$properties['itemContent'] = $itemContent;
	$properties['itemDescription'] = $itemDescription;
	
	if($itemContent != null) {
		echo $modx->getChunk('androidXmlTpl', $properties);
	}
}

echo "</items>";

45 Posts

Beniamin Reply #4, 13 years, 3 months ago

Well, I figured out my problem. There was a stray return call hiding in a block of code labelled "Tag processing" that I apparently didn't finish adapting to work in this context. When the script got to an item with no tags, it just quit the whole program.

☆ A M B ☆
427 Posts

W. Shawn Wilkerson Reply #5, 13 years, 3 months ago

You might also take a look at getIteration... Similar to getCollection, but lower on memory and is designed for, well, iteration.

Get your copy of MODX Revolution Building the Web Your Way http://www.sanitypress.com/books/modx-revolution-building-the-web-your-way.html

Check out my MODX || xPDO resources here: http://www.shawnwilkerson.com

45 Posts

Beniamin Reply #6, 13 years, 3 months ago

Thanks! That does help; I tried getIterator and it did make my script run faster.

5,888 Posts

Bruno17 Reply #7, 13 years, 3 months ago

All this getTVValue may also slow your script and running a getChunk in each loop can also need some time.

you can try

if ($modx->getObject('modTemplateVarResource',array('tmplvarid'=>$tvid,'contentid'=>$resid))){
    $value = $tv->get('value');
}

if you have output-filters in your chunk, this can also be an issue.
if not, you could do a simple str_replace, which should be faster than getChunk or you can try to collect all datas of each loop in placeholders, rename the placeholders in your chunk, while collecting them and do one big getChunk at the end.
With this technique I could speed parsing a lot in some situations.

I think there is some room to speed this thing up a bit.

-------------------------------

you can buy me a beer, if you like MIGX

http://webcmsolutions.de/migx.html

Thanks!