We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
MODX Community Forums Archive

Importing External Documents

    • 22827
    • 129 Posts
    Paul Reply #1, 11 years, 4 months ago
    Hello,

    I have a customer with an external document repository, a newsletter system. I am in the process of writing a snippet that will access the external documents via their API and display them through a modx site.

    It occurs to me that it would be far more convenient to import these documents, programatically, as modx resources. This way I can use all the usual snippets to display them (getResources, getPage etc) and also means the content is cached without me having to write my own caching. The documents never change once they are created, so all I need to do is get a list, compare it to what is already present, and add any new ones.

    Is creating MODx resources this way a sensible approach? I haven't explored the modx api before to do this type of thing. The unique id of each document on the remote site is numeric, so I could use these as the modx resource ID which would make comparison easy.

    Any pointers - if anyone has done anything similar and has examples, it would be great to see them.
      • 22827
      • 129 Posts
      Paul Reply #2, 11 years, 4 months ago
      Ok, I have found a few threads - no idea why I missed them first time around.
      $doc = $modx->newObject('modDocument');
...
$doc->save();


      Is it possible to set the id of the document before saving - after having checked it doesn't exist of course:

      $doc->set("id",$myid);


      (or similar), or is this internal to xpdo, and I have to use a resource field to store the external unique id?
        • 3749
        • 24,544 Posts
        BobRay Reply #3, 11 years, 4 months ago
        You could put the external IDs in an unused text field (e.g., introtext, description). You could also store them in a TV, but having them in a standard field will make retrieval quite a bit faster and easier.

        TBH, I'm not sure if you can specify the ID or not -- a quick test should tell you. If so, you could avoid collisions by adding some constant to all of them that's larger than your highest current ID (assuming that you're only doing this once).
          Did I help you? Buy me a beer
          Get my Book: MODX:The Official Guide
          MODX info for everyone: http://bobsguides.com/modx.html
          My MODX Extras
          Bob's Guides is now hosted at A2 MODX Hosting
        • benmarte Reply #4, 11 years, 4 months ago
          If you can get the api returning an xml or a JSON string you can just make 1 resource and have it display the fields you need and make a list with all the other documents in order to change documents, this is just me throwing out ideas IDK what you project looks like or it's needs.

          It is much easier to manage 1 resource that is dynamic than have MODX create hundreds of resources and try managing them.

          Just my 2 cents.
            Benjamin Marte
            Interactive Media Developer
            Follow Me on Twitter | Visit my site | Learn MODX
            • 22827
            • 129 Posts
            Paul Reply #5, 11 years, 4 months ago
            Quote from: BobRay at Jan 03, 2013, 08:57 PM
            You could put the external IDs in an unused text field (e.g., introtext, description). You could also store them in a TV, but having them in a standard field will make retrieval quite a bit faster and easier.

            TBH, I'm not sure if you can specify the ID or not -- a quick test should tell you. If so, you could avoid collisions by adding some constant to all of them that's larger than your highest current ID (assuming that you're only doing this once).

            Right, not using ID means that I don't need to worry about collisions, and I can test "just" as quickly if it is in a primary resource field:

            $test = $xpdo->getObject('modResource', array('description'=>$externalID));


            If this returns a null, I can add a new document, and let auto increment pick the id - this seems safer if I understand it correctly.
            • Everett Reply #6, 11 years, 4 months ago
              Along with benmarte, I would be concerned if the API is going to continue to be used -- in that case, it is the source for new content, and copying data into MODX resources could be problematic in the long run.

              Maybe you can write a snippet that utilizes its own cache -- i.e. it caches the data from each unique ID coming from the API. Just set it to never expire (or expire seldomly). Then you'd just have one Snippet to maintain and your site would act as a transparent window into the data in the API.
                Hosting stuff: http://fireproofsocks.com/
                Coding stuff: http://craftsmancoding.com/
                Blog stuff: http://tipsfor.us/
                • 22827
                • 129 Posts
                Paul Reply #7, 11 years, 4 months ago
                Quote from: Everettg_99 at Jan 04, 2013, 10:58 PM
                Along with benmarte, I would be concerned if the API is going to continue to be used -- in that case, it is the source for new content, and copying data into MODX resources could be problematic in the long run.

                What is concern about using an API over the long term, and what are the problems you can see arising from copying data into MODX resources?


                Maybe you can write a snippet that utilizes its own cache -- i.e. it caches the data from each unique ID coming from the API. Just set it to never expire (or expire seldomly). Then you'd just have one Snippet to maintain and your site would act as a transparent window into the data in the API.

                Yes, this is definitely an option, but the one I am hoping to avoid.
                • Everett Reply #8, 11 years, 3 months ago
                  What is concern about using an API over the long term, and what are the problems you can see arising from copying data into MODX resources?

                  The primary problem I'd be concerned about is duplicating data. Say for example, some of the content changes with the API provider (i.e. the master). You'd have to manually change all the pages in your MODx site -- that gets untenable very quickly. It's the same fundamental concern when normalizing database tables: you structure your database so that data lives in only one place, otherwise stuff gets very difficult to maintain.

                  If you called the API for all your content, you're letting it be the master -- it will have final say, and you're letting MODx be a transparent slave. And if you cache all the calls locally, you get all the speed benefits. If something changes @ the source (i.e. with the API provider), then you just simply clear your MODx cache and POOF, the new content trickles in without you having to do anything.

                  Hope that makes sense.
                    Hosting stuff: http://fireproofsocks.com/
                    Coding stuff: http://craftsmancoding.com/
                    Blog stuff: http://tipsfor.us/