We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 42141
    • 73 Posts
    Hi,

    I wanted to check my Modx-website for broken links with Xenu link checker, but he doesn't crawl my site for errors. I had Xenu listed as disallowed in my robots.txt so I removed those references. But still Xenu won't crawl the website. I immediately get a message that link check has completed and if I want to run a report. Is this just the cache for the robots.txt interfering and will that fix itself once that is cleared/updated, or is it something with Modx?

    Sophie
      • 36816
      • 109 Posts
      Greetings Sophie,

      I use Xenu to link-check MODX sites all the time with no problem.

      After scan completion, but before displaying the report, what is in the status column of Xenu report for the first (probably only, based on what you said) URL in its results list?

      Just a guess -- you may have missed something in robots.txt, or the server may be blocking based on Xenu's user agent. Rename the robots.txt for a minute and try Xenu again. If it runs, the cause is in there somewhere.

      If not and the results status column shows something like "Forbidden", it is possible that the server is waving Xenu off based on its user agent string. See #20 in Xenu's info page: http://home.snafu.de/tilman/xenulink.html

      If your server is running Apache, one place to look for such blocks is in the .htaccess file; check for entries such as "BrowserMatch" or "BrowserMatchNoCase". If there's nothing like that in there, and you're seeing "Forbidden" on Xenu's scan, you may be bumping up against a Web host placed block of Xenu's user agent. If so, contacting the hosting provider could get you some answers.

      Lastly, just for a lark, you may want to aim Xenu's crawl start at pages other than your root (if you've not done that) - it may provide some diagnostic information.
        • 42141
        • 73 Posts
        Hi Clareoconsulting,

        I start up Xenu, fill in the adress for my site and click on OK. I immediatly get a pop up that Link sleuth has finished and if I want to display a report. My domain name has the status of OK in the Xenu-screen. Renaming my robots.txt or .htaccess do not help. Crawling specific pages does not help either.

        It's not my webhost blocking Xenu, because I can scan my subdomains of that site with Xenu just fine, that why it is so strange.

        Sophie
          • 36816
          • 109 Posts
          Quote from: sophieschoice at Mar 30, 2014, 08:43 AM
          Crawling specific pages does not help either.

          What does the status column (just say "no" to the report) say when you start the crawl at specific pages?

          Quote from: sophieschoice at Mar 30, 2014, 08:43 AM
          Renaming my robots.txt or .htaccess do not help

          Renaming robots.txt shouldn't change the way the site works, but renaming .htaccess is very likely to make it not only not crawalable, but not work in general so I wouldn't assume yet that there's nothing in the .htaccess.

          Care to share the URL of the site that's not working for you with Xenu? Might help to look at it from elsewhere.
            • 42141
            • 73 Posts
            Quote from: clareoconsulting at Mar 30, 2014, 09:11 AM
            What does the status column (just say "no" to the report) say when you start the crawl at specific pages?
            Status: OK.

            Quote from: clareoconsulting at Mar 30, 2014, 09:11 AM

            Renaming robots.txt shouldn't change the way the site works, but renaming .htaccess is very likely to make it not only not crawalable, but not work in general so I wouldn't assume yet that there's nothing in the .htaccess.

            Care to share the URL of the site that's not working for you with Xenu? Might help to look at it from elsewhere.

            I will send you an PM smiley wink

              • 42141
              • 73 Posts
              This my .htacces:

              # MODX supports Friendly URLs via this .htaccess file. You must serve web
              # pages via Apache with mod_rewrite to use this functionality, and you must
              # change the file name from ht.access to .htaccess.
              #
              # Make sure RewriteBase points to the directory where you installed MODX.
              # E.g., "/modx" if your installation is in a "modx" subdirectory.
              #
              # You may choose to make your URLs non-case-sensitive by adding a NC directive
              # to your rule: RewriteRule ^(.*)$ index.php?q=$1 [L,QSA,NC]
              <IfModule mod_rewrite.c>
              RewriteEngine On
              RewriteBase /
              
              # Rewrite www.domain.com -> domain.com -- used with SEO Strict URLs plugin
              #RewriteCond %{HTTP_HOST} .
              #RewriteCond %{HTTP_HOST} !^example-domain-please-change\.com [NC]
              #RewriteRule (.*) http://example-domain-please-change.com/$1 [R=301,L]
              #
              # or for the opposite domain.com -> www.domain.com use the following
              # DO NOT USE BOTH
              #
              RewriteCond %{HTTP_HOST} .
              RewriteCond %{HTTP_HOST} !^www\.exampledomain\.nl [NC]
              RewriteRule (.*) http://www.exampledomain.nl/$1 [R=301,L]
              
              # The Friendly URLs part
              RewriteCond %{REQUEST_FILENAME} !-f
              RewriteCond %{REQUEST_FILENAME} !-d
              RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]
              </IfModule>


              I have no clue why Xenu can't crawl my domain, yet it can crawl the subdomains (Wordpress-sites). So it isn't blocked by my host...

                • 42141
                • 73 Posts
                Anyone?
                  • 28042 ☆ A M B ☆
                  • 24,524 Posts
                  I don't know anything about Xenu, but that .htaccess file can't work.
                  RewriteCond %{HTTP_HOST} .
                  RewriteCond %{HTTP_HOST} !^www\.exampledomain\.nl [NC]
                  RewriteRule (.*) http://www.exampledomain.nl/$1 [R=301,L]
                  

                  I am presuming your domain is not exampledomain.nl
                    Studying MODX in the desert - http://sottwell.com
                    Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
                    Join the Slack Community - http://modx.org
                    • 42141
                    • 73 Posts
                    I anonymized the domain name in the post here wink My real .htaccess does include my domain name.
                      • 28042 ☆ A M B ☆
                      • 24,524 Posts
                      Have you tried it on any other domains using MODX?
                        Studying MODX in the desert - http://sottwell.com
                        Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
                        Join the Slack Community - http://modx.org