We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 37059
    • 368 Posts
    I have a site, which we’ll call samplewidgets.com, where each widget has an associated page, and on each page, there is a link to download a PDF about that widget. The URLs are formed thusly:

    http://www.samplewidgets.com/widgets/green
    http://www.samplewidgets.com/assets/files/green.pdf

    In practice, this works fine. This morning, however, Notify404 sent me a bunch of errors encountered by what was apparently a bot. The bad URLs looked like this:

    http://www.samplewidgets.com/widgets/assets/files/green.pdf

    I am assuming that this is due to my use of relative pathnames when linking to the PDFs, e.g.

    assets/files/green.pdf

    So hopefully I have fixed the problem by adding [[++site_url]] in front of each link.

    My question is, should this be happening and will what I have done fix it? The bot that got 404’d this morning wasn’t Google, but I certainly don’t want a major search engine seeing a site full of not-found errors!
      Jason
      • 18373 ☆ A M B ☆
      • 3,141 Posts
      I get the same, too.

      I’m running a slightly updated version of Notify404 (which I still need to wrap up) which gives more info on the visitor.

      Under your email details, replace all of it with this:
        /* Unless you know PHP and want to modify this plugin, do not edit below */
        /*************************************************************************/
        
        
        if ($mailto == '') { 
          $modx->log(modX::LOG_LEVEL_ERROR,'Error: mailto not specified for Notify404');
          return;
        }
        
        $phs['ip'] = $_SERVER['REMOTE_ADDR'];
        $phs['host'] = 'http'. ($_SERVER['HTTPS'] ? 's' : null) .'://'. $_SERVER['HTTP_HOST'];
        $phs['request'] = $_SERVER['REQUEST_URI'];
        $phs['timestamp'] = date('d M Y - G:i:s');
        $phs['referer'] = ($_SERVER['HTTP_REFERER']) ? $_SERVER['HTTP_REFERER'] : 'Not available.';
        $phs['user_agent'] = ($_SERVER['HTTP_USER_AGENT']) ? $_SERVER['HTTP_USER_AGENT'] : 'Not available.';
      
        if (strstr($phs['request'],'reflect')) { return; }
        
        $message = $modx->getChunk($mailchunk,$phs);
        if ($message == '') {
          $message = '<p>This is an automated email notification of a 404 error that has occured on the [[++site_name]] website.<br /><br />
      
      <strong>Date & time</strong>: [[+timestamp]]<br />
      <strong>Requested</strong>: [[+request]]<br />
      <strong>Host</strong>: [[+host]]<br />
      <strong>Visitor IP</strong>: [[+ip]]<br />
      <strong>Referer</strong>: [[+referer]]<br />
      </strong>User agent</strong>: [[+user_agent]]<br /><br />
      
      These automatic notifications can be disabled by disabling the Notify404 plugin in your [[++site_name]] manager.<br />
      Sent by Notify404 from [[++site_url]]</p>
      ';
        }
      
        $modx->getService('mail', 'mail.modPHPMailer');
        $modx->mail->set(modMail::MAIL_BODY,$message);
        $modx->mail->set(modMail::MAIL_FROM,$mailfrom);
        $modx->mail->set(modMail::MAIL_FROM_NAME,'Notify404');
        $modx->mail->set(modMail::MAIL_SENDER,'Notify404');
        $modx->mail->set(modMail::MAIL_SUBJECT,'[Notify404] Not found: '.$phs['request']);
        $modx->mail->address('to',$mailto);
        $modx->mail->address('reply-to',$replyto);
        $modx->mail->setHTML(true);
        if (!$modx->mail->send()) {
            $modx->log(modX::LOG_LEVEL_ERROR,'An error occurred while trying to send the email: '.$err);
        }
        $modx->mail->reset();​


      It will also send you the user agent and some other details like the referer. Oh and I added in a check to not send me emails of failed attempts to hack my site through an Evo exploit (reflect one).. my site is targetted often due to the "modx" in some of the urls.

      What I find is that most of the 404’s are from java user agents who don’t understand relative urls (not even with a proper base href)...
        Mark Hamstra • Developer spending his days working on Premium Extras and a MODX Site Dashboard with the ability to remotely upgrade MODX and extras to make the MODX world a little better.

        Tweet me @mark_hamstra, check my infrequent blog at markhamstra.com, my slightly more frequent ramblings at MODX.today or see code at Github.
        • 37059
        • 368 Posts
        Thanks for the quick and enlightening reply, Mark. I hope (assume?) the Big 3 bots are smart enough to handle relative paths, at least when they’re spelled out as I have them now!

        I have updated my copy of the plugin per your instructions. How does the new code address the problem?

        Thanks again.
          Jason
          • 18373 ☆ A M B ☆
          • 3,141 Posts
          It doesn’t address the problem.

          Just gives you more info in the email to find out who/what/why is causing the issue.

          :P
            Mark Hamstra • Developer spending his days working on Premium Extras and a MODX Site Dashboard with the ability to remotely upgrade MODX and extras to make the MODX world a little better.

            Tweet me @mark_hamstra, check my infrequent blog at markhamstra.com, my slightly more frequent ramblings at MODX.today or see code at Github.
            • 37059
            • 368 Posts
            Quote from: Mark at Mar 01, 2011, 09:53 AM

            It doesn’t address the problem.
            LOL grin

            Here’s an idea: Why not have a list which we could use to filter errors from certain IP addresses/user agents, i.e. those which make a habit of misinterpreting URLs? Seems like that wouldn’t be too hard to implement.
              Jason
              • 18373 ☆ A M B ☆
              • 3,141 Posts
              When I got time on my hands, I want to extend Notify404 with a CMP which lists all attempts, errors not to send emails for, user agents to ignore etc.

              Don’t have a lot of time though. tongue

              (unless someone wants to move this up in the priority list with a bounty)
                Mark Hamstra • Developer spending his days working on Premium Extras and a MODX Site Dashboard with the ability to remotely upgrade MODX and extras to make the MODX world a little better.

                Tweet me @mark_hamstra, check my infrequent blog at markhamstra.com, my slightly more frequent ramblings at MODX.today or see code at Github.
                • 37059
                • 368 Posts
                Unfortunately I’ve had to disable Notify404 out of sheer annoyance - not with the plugin smiley but with the bots! After I added the absolute pathname to all my resource URLs using [[+site_url]], e.g. samplewidgets.com/assets/files/green.pdf, I started getting bot-triggered 404’s like this:

                /assets/files/green.pdf

                >:( I’m not sure how simple this must be made before the bots will be able to figure it out! I suppose I could use robots.txt to block the offenders, but I wouldn’t want to inadvertently block something important.

                Thanks again for your effort of developing the plugin Mark. It’s not quite worth a bounty wink to me, but I will await any further enhancements for which you may find time in the future.
                  Jason