Parse PDF content for search results

45 Posts

matthewmeredith Reply #11, 7 years, 10 months ago

*** VERY IMPORTANT UPDATE ***

Anyone using SimpleSearch.Solr, be warned:

SOLR NO LONGER USES SCHEMA.XML. The SimpleSearch documentation NEEDS to be updated. I spent a good 3-4 days trying to figure this bloody thing out and all along my changes to schema.xml were doing absolutely nothing! It now uses managed-schema. Wish I had known that from the start...

Anyways, rant over. There are still lots of issues with the SimpleSearch.Solr. Using the solr.logs, I've found quite a few problems in the query that actually gets sent to Solr which results in zero results. For example (from simplesearchdriversolr.class.php):

        /* add fields to search
    	$fields = $this->modx->getFields('modResource');
        foreach ($fields as $fieldName => $default) {
            $query->addField($fieldName);
        } */

Generates a HUGE list of fields to search, and when these aren't properly shown in the managed-schema (which they weren't), you get the "Error running query on Solr server: undefined field deleted" I was getting earlier. Totally baffled me because the field "deleted" was in my schema.xml but NOT in my managed-schema.

Moving on.

I can now run a search and end up with no error messages (in MODX or solr.log) but get zero results. The generated query looks like this:

webapp=/solr path=/select params={q=parks&indent=on&start=0&fq=published:1&fq=searchable:1&fq=deleted:0&rows=10&version=2.2&wt=xml} hits=0 status=0 QTime=1

Trying to simplify even more, I wanted to get rid of the "fq" values (filter query). So I commented out the entire following section (again from simplesearchdriversolr.class.php):

        /* handle hidemenu option */
/*      
        $hideMenu = $this->modx->getOption('hideMenu',$scriptProperties,2);
        if ($hideMenu != 2) {
            $query->addFilterQuery('hidemenu:'.($hideMenu ? 1 : 0));
        }
*/

        /* handle contexts */
/*
        $contexts = $this->modx->getOption('contexts',$scriptProperties,'');
        $contexts = !empty($contexts) ? $contexts : $this->modx->context->get('key');
        $contexts = implode(' ',explode(',',$contexts));
    	$query->addFilterQuery('context_key:('.$contexts.')');
*/

        /* handle restrict search to these IDs */
/*
        $ids = $this->modx->getOption('ids',$scriptProperties,'');
    	if (!empty($ids)) {
            $idType = $this->modx->getOption('idType',$this->config,'parents');
            $depth = $this->modx->getOption('depth',$this->config,10);
            $ids = $this->processIds($ids,$idType,$depth);
            $query->addFilterQuery('id:('.implode(' ',$ids).')');
        }
*/

        /* handle exclude IDs from search */
/*
        $exclude = $this->modx->getOption('exclude',$scriptProperties,'');
        if (!empty($exclude)) {
            $exclude = $this->cleanIds($exclude);
            $exclude = implode(' ',explode(',', $exclude));
            $query->addFilterQuery('-id:('.$exclude.')');
        }
*/

        /* basic always-on conditions */
/*
        $query->addFilterQuery('published:1');
        $query->addFilterQuery('searchable:1');
        $query->addFilterQuery('deleted:0');
*/

Now I'm getting an error in my browser stating:

Fatal error: Call to a member function checkPolicy() on a non-object in /home/townofco/core/components/simplesearch/model/simplesearch/driver/simplesearchdriversolr.class.php on line 220

and a MODX error log of:

[2016-06-18 02:49:10] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 595) PHP warning: strrpos() expects parameter 1 to be string, array given
[2016-06-18 02:49:10] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 600) PHP warning: strrpos() expects parameter 1 to be string, array given
[2016-06-18 02:49:10] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 606) PHP warning: strtolower() expects parameter 1 to be string, array given
[2016-06-18 02:49:10] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 617) PHP warning: Illegal offset type in isset or empty
[2016-06-18 02:49:10] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 618) PHP warning: class_exists() expects parameter 1 to be string, array given
[2016-06-18 02:49:10] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 643) Could not load class: Array from mysql..

BUT I'm getting hits in Solr finally (from solr.log):

webapp=/solr path=/select params={q=parks&indent=on&start=0&rows=10&version=2.2&wt=xml} hits=30 status=0 QTime=0

I know nothing about PHP so I'm back to being at a total loss. However, I still feel that this is a step in the right direction. Is anyone able to get the missing puzzle piece?!

Edit: I commented out the little chunk in simplesearchdriversolr.class.php that was giving me the error message:

/** @var modResource $resource */
                       /*  $resource = $this->modx->newObject($d['class_key']);
                        if ($resource->checkPolicy('list')) {
                            $response['results'][] = $d;
                        } */

and I don't get any error messages in MODX. A search results page still shows zero results. However, in my solr.log I am still getting hits:

webapp=/solr path=/select params={q=parks&indent=on&start=0&rows=10&version=2.2&wt=xml} hits=30 status=0 QTime=0

Am I correct in thinking that piece that I commented out is what generates the results array? If so, how can I put it back in without the previous error messages? [ed. note: matthewmeredith last edited this post 7 years, 10 months ago.]

24,544 Posts

BobRay Reply #12, 7 years, 10 months ago

It should work to do this (assuming that the class_key is correct), though the policy won't be checked and resources the user doesn't have access to may show up (if there are any):

$resource = $this->modx->newObject($d['class_key']);
if ($resource) {
    $response['results'][] = $d;
}

Did I help you? Buy me a beer
Get my Book: MODX:The Official Guide
MODX info for everyone: http://bobsguides.com/modx.html
My MODX Extras
Bob's Guides is now hosted at A2 MODX Hosting

45 Posts

matthewmeredith Reply #13, 7 years, 10 months ago

Quote from: BobRay at Jun 18, 2016, 06:04 AM

It should work to do this (assuming that the class_key is correct), though the policy won't be checked and resources the user doesn't have access to may show up (if there are any):
$resource = $this->modx->newObject($d['class_key']);
if ($resource) {
    $response['results'][] = $d;
}

Thanks Bob, that's sort of what I was thinking but I don't know how to properly do it in PHP. I tried your suggestion (as I don't have any private resources) but it's still not working. I'm getting zero results shown on the front-end, but Solr is still getting hits in solr.log. Here's my MODX error log:

[2016-06-18 15:34:22] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 595) PHP warning: strrpos() expects parameter 1 to be string, array given
[2016-06-18 15:34:22] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 600) PHP warning: strrpos() expects parameter 1 to be string, array given
[2016-06-18 15:34:22] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 606) PHP warning: strtolower() expects parameter 1 to be string, array given
[2016-06-18 15:34:22] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 617) PHP warning: Illegal offset type in isset or empty
[2016-06-18 15:34:22] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 618) PHP warning: class_exists() expects parameter 1 to be string, array given
[2016-06-18 15:34:22] (ERROR @ /home/townofco/core/xpdo/xpdo.class.php : 643) Could not load class: Array from mysql..

and it repeats itself several times.

Any thoughts?

Edit: What is 'class_key'? Is it just the Resource Type? Can I change that to

$resource = $this->modx->newObject($d['Document']);

if I want it to only search through Resources of the Document type? [ed. note: matthewmeredith last edited this post 7 years, 10 months ago.]

24,544 Posts

BobRay Reply #14, 7 years, 10 months ago

That's xPDO trying (and failing) to load the class specified in the class_key.

You might try:

$resource = $this->modx->newObject('modResource');

or

$resource = $this->modx->newObject('modDocument');

Did I help you? Buy me a beer
Get my Book: MODX:The Official Guide
MODX info for everyone: http://bobsguides.com/modx.html
My MODX Extras
Bob's Guides is now hosted at A2 MODX Hosting

45 Posts

matthewmeredith Reply #15, 7 years, 10 months ago

Quote from: BobRay at Jun 18, 2016, 08:22 PM

That's xPDO trying (and failing) to load the class specified in the class_key.

You might try:
$resource = $this->modx->newObject('modResource');
or
$resource = $this->modx->newObject('modDocument');

Bob, you're a legend!!! Okay, now when I search for "bylaw", I get my SimpleSearch page displaying "27 results found for 'bylaw'", 3 pages of pagination, and a list from 1 to 10.... But no contents!!! I'm so happy though; this is the closest I've been to having this work. By the way, the 27 results found matches my Solr core perfectly when I run the query in Solr Admin.

Here's my new MODX error log:

[2016-06-19 21:03:49] (ERROR @ /home/townofco/core/model/modx/modx.class.php : 1984) PHP warning: Illegal offset type in isset or empty
[2016-06-19 21:03:49] (ERROR @ /home/townofco/core/model/modx/modx.class.php : 1985) PHP warning: Illegal offset type
[2016-06-19 21:03:49] (ERROR @ /home/townofco/core/model/modx/modx.class.php : 1986) PHP warning: Illegal offset type
[2016-06-19 21:03:49] (ERROR @ /home/townofco/core/model/modx/modx.class.php : 1990) PHP warning: Illegal offset type
[2016-06-19 21:03:49] (ERROR @ /home/townofco/core/components/simplesearch/model/simplesearch/simplesearch.class.php : 296) PHP warning: strip_tags() expects parameter 1 to be string, array given

Again, repeating 10 times.

ADDITION: I re-indexed my PDF files that I need parsed for content, and when I use the search term "hemerocailis" (of which I know there will only be one result from the PDF's content), I get the following MODX error message:

[2016-06-19 21:25:43] (ERROR @ /home/townofco/core/model/modx/modx.class.php : 983) `/home/townofco/public_html/modx/assets/pdfs/DeerResistantPlants1.pdf` is not a valid integer and may not be passed to makeUrl()

[ed. note: matthewmeredith last edited this post 7 years, 10 months ago.]

45 Posts

matthewmeredith Reply #16, 7 years, 10 months ago

Thanks to samtux on the #modx freenode channel, the strip_tags() error is gone and there are bits of text in my search results! We added the line:

$text = is_array($text) ? implode($text) : $text;

to the start of the function so it is now:

    public function sanitize($text) {
	$text = is_array($text) ? implode($text) : $text;
        $text = strip_tags($text);
        $text = preg_replace('/(\[\[\+.*?\]\])/i', '', $text);
        return $this->modx->stripTags($text);
    }

There is now bits of content in the search results (for Resources only)! No links, and nothing for PDF results... But progress is progress laugh

MODX errors are now:

[2016-06-19 23:28:48] (ERROR @ /home/townofco/core/model/modx/modx.class.php : 1984) PHP warning: Illegal offset type in isset or empty
[2016-06-19 23:28:48] (ERROR @ /home/townofco/core/model/modx/modx.class.php : 1985) PHP warning: Illegal offset type
[2016-06-19 23:28:48] (ERROR @ /home/townofco/core/model/modx/modx.class.php : 1986) PHP warning: Illegal offset type
[2016-06-19 23:28:48] (ERROR @ /home/townofco/core/model/modx/modx.class.php : 1990) PHP warning: Illegal offset type

for Resource hits, and:

[2016-06-19 23:28:48] (ERROR @ /home/townofco/core/model/modx/modx.class.php : 983) `/home/townofco/public_html/modx/assets/pdfs/minutes and agendas/2015Minutes/08-19RCMMinutes.pdf` is not a valid integer and may not be passed to makeUrl()

for PDF hits.

Thanks Bob and Sam for your help! Hopefully we can crush this out and have a solution for Solr and Simple Search that everyone can use!

45 Posts

matthewmeredith Reply #17, 7 years, 10 months ago

Okay, I got rid of the Illegal offset type errors by changing line 97 in my SimpleSearch snippet from:

$resourceArray['link'] = $modx->makeUrl($resourceArray['id'],'$ctx',$args);

to:

$resourceArray['link'] = $modx->makeUrl($resourceArray['id'],'',$args);

As I am really not concerned about Context so I'm just not going to try to grab that.

Now the question is... How do I get my PDFs that are being returned by Solr as results (which is great!!!) to turn into URLs... Hopefully with makeUrl somehow? Is there some sort of PHP I can add to the SimpleSearch snippet that will use the above^^ makeUrl code ONLY when the result is a Resource, and will just link to the PDF otherwise?

EDIT:

I undid what I did above as so I'm back to:

$resourceArray['link'] = $modx->makeUrl($resourceArray['id'],'$ctx',$args);

I have been going through all my steps and making sure I actually needed to do them. I went a LONG ways before I found out that Solr uses managed-schema instead of schema.xml as per the documentation. I had commented out a bunch of filterQuery sections of my simplesearchdriversolr.class.php, one of which had to do with contexts.

I am now error-free except for makeUrl trying to use the path to my PDF assets instead of an ID. Yay! Hopefully someone can help getting that last bit sorted! [ed. note: matthewmeredith last edited this post 7 years, 10 months ago.]

45 Posts

matthewmeredith Reply #18, 7 years, 10 months ago

WARNING: TOTAL PHP NOOB ALERT!

Okay, I've added this to my SimpleSearch snippet:

if ($resourceArray['context_key'] = 'web') {
            
            
            $resourceArray['link'] = $modx->makeUrl($resourceArray['id'],'$ctx',$args);
            } else {
                /* NO IDEA WHAT TO PUT HERE */
            }

... And I didn't get any errors! And my search results have stayed the same... Which is good. I *think* that whatever I put into the else statement will be applied to PDF results... But I'm not sure, nor am I sure how to test it.

Now, I just need to know what to put into that else statement! I'm assuming makeUrl won't work as the url needs to point directly to the PDF file, not an ID. It needs to be something like:

$resourceArray['link'] = LINK_TO_PDF_IN_ASSET_FOLDER;

but I have no idea how to do that... Can anyone help?

EDIT: Well, that didn't work... it went into the "if" statement for PDF results as well... Okay, I have to figure out a better "if" condition that will only return resources, and then I can figure out how to generate URLs to PDFs. [ed. note: matthewmeredith last edited this post 7 years, 10 months ago.]

45 Posts

matthewmeredith Reply #19, 7 years, 10 months ago

OMG I actually made it work! So much trial and error, but it does the trick.

/* get results */
$response = $search->getSearchResults($searchString,$scriptProperties);
$placeholders = array('query' => $searchString);
$resultsTpl = array('default' => array('results' => array(),'total' => $response['total']));
if (!empty($response['results'])) {
    /* iterate through search results */
    foreach ($response['results'] as $resourceArray) {
        $resourceArray['idx'] = $idx;
        if (empty($resourceArray['link'])) {
            $ctx = !empty($resourceArray['context_key']) ? $resourceArray['context_key'] : $modx->context->get('key');
            $args = '';
            if ($addSearchToLink) {
                $args = array($searchInLinkName => $searchString);
            }  
        }
        if ($showExtract) {
            $extract = $searchString;
            if (array_key_exists($extractSource, $resourceArray)) {
                $text = $resourceArray[$extractSource];
            } else {
                $text = $modx->runSnippet($extractSource, $resourceArray);
            }
            $extract = $search->createExtract($text,$extractLength,$extract,$extractEllipsis);
            /* cleanup extract */
            $extract = strip_tags(preg_replace("#\<!--(.*?)--\>#si",'',$extract));
            $extract = preg_replace("#\[\[(.*?)\]\]#si",'',$extract);
            $extract = str_replace(array('[[',']]'),'',$extract);
            $resourceArray['extract'] = !empty($highlightResults) ? $search->addHighlighting($extract,$highlightClass,$highlightTag) : $extract;
        }
        

        /*THIS IS THE PART I CHANGED AND MOVED */
        if ($extract) {
                /* KEEP THIS LINE AS IT IS THE ORIGINAL CODE */
                $resourceArray['link'] = $modx->makeUrl($resourceArray['id'],'$ctx',$args);
                
            } else {
                /* NO IDEA WHAT TO PUT HERE */
                $resourceArray['link'] = 'http://google.ca';
            }
        
        $resultsTpl['default']['results'][] = $search->getChunk($tpl,$resourceArray);
        $idx++;
    }
}

It's definitely a hack, but it's generating the correct makeUrl for Resource hits and filling in http://google.ca for PDF hits. Okay, just have to generate the URL to the PDF assets now....... Help?

45 Posts

matthewmeredith Reply #20, 7 years, 10 months ago

I DID IT!!! Wow, so happy right now. I actually think it's quite the elegant solution, too!

if ($extract) {
  /* KEEP THIS LINE AS IT IS THE ORIGINAL CODE */
  $resourceArray['link'] = $modx->makeUrl($resourceArray['id'],'$ctx',$args);
  /* SOME WAY TO GET TITLE OF PAGE */
  /* $resourceArray['pagetitle'] = $page->get('pagetitle'); */

} else {
  /* GENERATE LINKS TO PDF HITS */
  $resourceArray['link'] = substr($resourceArray['id'], 26);
  /* SOME WAY TO GET TITLE OF PDF */
}

So I noticed that the error I was getting with a string being passed to makeUrl was the string I was needing! So I just grabbed that on its own and then stripped the first 26 characters to get rid of the /home/townofco/public_html/

SO EXCITED!

Now, to somehow pass titles to mySearchResult tpl to get [[+pagetitle]] and [[+longtitle]] working......