Sorry: finding and indexing pdf, that’s quite another cup of tea
At first I’d like to have the best way to help indexer finding all relevant documents/files (all html and pdf). There are 2 (separate!) ways finding docs with Sphider: crawling the whole site OR (logically XOR) using the sitemap.xml. Using sitemap increases the indexing performance because of the predefined paths to all relevant docs. Sphider follows only all path within the sitemap and doesn’t search for other paths in the site.
Two issues now:
- Building the sitemap I can’t input the paths to pdf files.
- Regular crawling the site Sphider DOESN’T find all paths. There is a bug (or a feature
) with relative paths. A path formed as
beginning with leading slash will be followed by spider. Paths without the leading slash are ignored (and I don’t know why). Therefore I’d prefer to use the sitemap.
Second: depending on server configuration and/or configuration of Sphider it indexes pdf files using "pdftotext". Sphider-plus has an included pdftotext converter as an exe. This doesn’t work on linux servers (on linux exe-files can’t executed). The other way is using precompiled version of pdftotext contained in the xpdf bundle. But there is (maybe only for me yet) a problem with the path. It doesn’t work this time and I’m searching for a solution. Nevertheless this is not the question here.
Return to start: is there a way to integrate paths to defined files into the sitemap?
... and returning to second issue: is there an easy way to run pdftotext (as a pre-compiled binary) on a shared linux host? I’m not a server specialist therefore I’m working by trial and error
MODX Staff