We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 26016
    • 561 Posts
    Hello Everyone,
    I have a MODx application up and running in prototype form. What a great framework!

    However, I’m not a database design guru. Right now I have only added one new table to MODx to get my application working, and I suspect that this isn’t an especially efficient design. It contains about 15 fields.

    I’ve been reading up on database normalization, and I’ve identified places where redundancy of data could occur. If I follow these rules to the letter, I could end up with about 5 related tables instead of one. I definitely understand how using tables in this manner can make queries faster if using a huge database or doing many types of queries.

    I foresee having maybe 1000 records or so maximum, and I’m trying to decide if re-designing the database is worthwhile. So far I only have a couple types of queries. There are also considerations like updating/storing data, vs. querying it.

    I understand that you can’t comment on my exact situation, but can you share your thoughts about what you have done in similar situations? What has worked (or not worked) for you?

    Thanks!!!!!!
      MODx and Wordpress development
      Linux, PHP 5.2, MySQL 5.0, Evo 1.05, Revo 2.08-pl, Firefox 4
      • 27226
      • 186 Posts
      Quote from: samba at Aug 23, 2006, 12:19 PM

      I’ve been reading up on database normalization, and I’ve identified places where redundancy of data could occur. If I follow these rules to the letter, I could end up with about 5 related tables instead of one. I definitely understand how using tables in this manner can make queries faster if using a huge database or doing many types of queries.
      As far as I know, normalization is not done for better performance, but for a more consistent handling of the data.
      (Basically, in terms of db-design, redundancy is not something that "could" occur - it is there or it isn’t.)
      The more "normalized" your DB is, the more tables you usually have to join in a query. And this is the aspect which can be costly in terms of performance. The "pro" starts with a normalized version of the db and then checks, where some "de-normalization" might lead to performance improvements.
      However: with just 15 fields and 1000 recs I wouldn’t bother so much with normalization at all, but make the structure as convenient as possible for interfacing it (from MODx/PHP), i.e. having simpler queries due to the absence of joins, unions and subqueries.
        • 26016
        • 561 Posts
        Quote from: silent at Aug 23, 2006, 01:42 PM

        As far as I know, normalization is not done for better performance, but for a more consistent handling of the data.

        The more "normalized" your DB is, the more tables you usually have to join in a query. And this is the aspect which can be costly in terms of performance. The "pro" starts with a normalized version of the db and then checks, where some "de-normalization" might lead to performance improvements.
        However: with just 15 fields and 1000 recs I wouldn’t bother so much with normalization at all, but make the structure as convenient as possible for interfacing it (from MODx/PHP), i.e. having simpler queries due to the absence of joins, unions and subqueries.

        Silent,
        Good points, thanks for the ideas. This was the direction I was leaning in. It’s clearly true that having queries joining 5 tables gets a little gnarly. And joining 5 tables sure seems as though it could be slow.

        Come to think of it, it would be funny to see who among us has the table with the largest number of fields! Maybe somebody could start a contest or MODx Olympics or something! grin

        Depending on how the database is used, I can see situations where normalization may actually help performance. Say you have a huge number of records. If you have a query that only goes after one 3-field table, as opposed to going after a giant 25-field table, disk reads should be faster, no?

        (Basically, in terms of db-design, redundancy is not something that "could" occur - it is there or it isn’t.)
        Maybe I’m a little off-track, but here’s a thought. Let’s say my one monster table contains a field called State. My thought was that I may have only one record containing "Alaska", even though it would be possible to have many. Wouldn’t that be a situation where redundancy may or may not occur? Sorry to split hairs. wink

        I appreciate the help!
          MODx and Wordpress development
          Linux, PHP 5.2, MySQL 5.0, Evo 1.05, Revo 2.08-pl, Firefox 4
          • 22303 MODX Staff
          • 10,725 Posts
          Quote from: samba at Aug 23, 2006, 02:35 PM

          Good points, thanks for the ideas. This was the direction I was leaning in. It’s clearly true that having queries joining 5 tables gets a little gnarly. And joining 5 tables sure seems as though it could be slow.

          ...

          Depending on how the database is used, I can see situations where normalization may actually help performance. Say you have a huge number of records. If you have a query that only goes after one 3-field table, as opposed to going after a giant 25-field table, disk reads should be faster, no?
          Absolutely. Scalability and performance is a balancing act, and database normalization definitely weighs in.

          In general, joining by foreign keys to normalized dependent tables does not necessarily increase or decrease performance, it completely depends on what kind of and quantity of data is being stored and retrieved. And though normalization does make transaction handling a little more complex, it generally increases insert/update performance, and improves your ability to get at specific parts of the data more quickly, and only when you need it. This can reduce memory footprint and increase query performance in certain situations where granular views of the individual, normalized tables would be useful.

          Another consideration in normalization is maintainability and extensibility. Ask yourself if this is something you’re going to use as-is forever, or if there other features you may want to add in the future that could benefit from normalization? One thing you don’t want to deal with is migrating data to a new, normalized (or de-normalized for that matter) model because you have to change it later on, especially if you have a lot of data. It’s possible the migration is worth the extra milliseconds of performance you gained until the new features were introduced, but at least in more complex data models, this can be a real pain in the rear.

          Just keep in mind that designing a web application requires you to consider the specific requirements of your application against the extremes of under-normalization and over-normalization of your data model. I for one tend toward normalization in most cases, but I’ve worked as a database administrator for several major corporations, managing, migrating, importing, and exporting every imaginable kind of sales and marketing data, so my experience helps guide me through the decisions. It’s a daunting task, designing a database, but can be one of the most important aspects of your web application, so I encourage you to continue to research the subject, and even experiment with different structures to see how they actually fare.

          Quote from: samba at Aug 23, 2006, 02:35 PM

          (Basically, in terms of db-design, redundancy is not something that "could" occur - it is there or it isn’t.)
          Maybe I’m a little off-track, but here’s a thought. Let’s say my one monster table contains a field called State. My thought was that I may have only one record containing "Alaska", even though it would be possible to have many. Wouldn’t that be a situation where redundancy may or may not occur? Sorry to split hairs. wink
          This is just another one of those decisions where it depends on how often the data is going to change. IMO, unless you are going to be internationalizing your site, data that doesn’t change very often, like states, is better maintained in code (i.e. a PHP array representing a normalized table) or even just in the HTML markup (i.e. form select options) and saved by name. And you can easily migrate such a structure to a normalized one later by simply adding a new fk column to the new table, and setting the values based on the unique set of values you store in your new table. Remove the old column, and bam!, instant normalization. But the performance benefits of storing state un-normalized, since it’s something that will only be changed (or managed) rarely, are obvious, if not trivial.
            • 647
            • 8 Posts
            In my experience the usabilty and maintainability of joined db-tables may depend on the CMS you are working with. In typo3  - the CMS I am using most - it is quite easy to integrate a single custom table. Appropriate forms to fill in the data are automatically generated for backend (=manager) users. But the maintainance of joined tables is less comfortable.
            For example: You want to manage addresses and seperate the cities, states and countries in joined tables. Now you have to fill in a new address of someone but your cities-, states- and countries-tables do not have the appropiate entries yet. In that situation you have first to fill in these data in the joint tables before you can add your address. In many cases - and not only with typo3 - this is quite cumbresome.
            To decide upon the oppropriate degree of normalization is a question of common sense, experience and task analysis, I think. Would you put the first-names of people in an address-db into a joint table to avoid redundancy? Probably not, even though there might be lots of Michals and Jessicas, Christophers and Ashleys... As OpenGeek observed, finally the decision about your data structure depends on the context and present and prospective future data usage.

            @samba: I am quite interested to learn, how you added a custom table to MODx for use in an application. How do you add data to this table? As a MODx newbie this is the point I am most interesed in.
              • 26016
              • 561 Posts
              Jason,
              Thanks for the excellent lesson! I’ve been reading up on this, but it’s especially useful to get opinions from a guru who is doing this every day. I’m more of an apps programmer, so I’ve simply used databases rather than designing them. Today I’m going to dig into your MakeTable and MakeForm code! I’m thinking that might be more fruitful than continuing to modify the Datagrid, useful though it may be.

              @samba: I am quite interested to learn, how you added a custom table to MODx for use in an application. How do you add data to this table? As a MODx newbie this is the point I am most interesed in.
              Aguen, that’s interesting about Typo. I used PHPMyAdmin to name, create, and modify my table. Depending on your hosting, you may have access to that already, or you could install it. As far as something inside MODx... I didn’t find a module or snippet that would help make tables, but somebody may have one, who knows? With all the fancy apps that people are making with MODx, I assume that somebody’s gotta have a slicker way to do this than PHPMyAdmin!

              I even used PHPMyAdmin to enter my initial bits of test data, but obviously it would be cumbersome to enter all data that way. So my to-do list today is to make a form that will facilitate data entry. Looking through the repository, I found "FormProcessor", which may save me some work:
              http://modxcms.com/FormProcessor-798.html

              Thanks, everybody!!!!
                MODx and Wordpress development
                Linux, PHP 5.2, MySQL 5.0, Evo 1.05, Revo 2.08-pl, Firefox 4
                • 647
                • 8 Posts
                Quote from: samba at Aug 24, 2006, 09:46 AM

                I even used PHPMyAdmin to enter my initial bits of test data, but obviously it would be cumbersome to enter all data that way. So my to-do list today is to make a form that will facilitate data entry. Looking through the repository, I found "FormProcessor", which may save me some work:
                http://modxcms.com/FormProcessor-798.html

                Samba, thanks for this link - i will have a look at it.
                  • 22303 MODX Staff
                  • 10,725 Posts
                  Quote from: samba at Aug 24, 2006, 09:46 AM

                  Jason,
                  Thanks for the excellent lesson! I’ve been reading up on this, but it’s especially useful to get opinions from a guru who is doing this every day. I’m more of an apps programmer, so I’ve simply used databases rather than designing them. Today I’m going to dig into your MakeTable and MakeForm code! I’m thinking that might be more fruitful than continuing to modify the Datagrid, useful though it may be.

                  Just FYI, if you take a look at http://xpdo.org/, this OR/M project which I started a few months ago, is a system for managing any persistent data, including complex foreign key relationships, auto-generating tables on first access, etc. I’m using it now exclusively for all custom db work I do with MODx, including creating an OO-based alternative API for 0.9.x. It will also power the 1.0 data model, as this will allow us to easily port MODx to other db platforms besides MySQL. I also have new versions of MakeTable and MakeForm available as part of this project, which can auto-build tables and forms from objects in addition to the approach of using associative arrays as they do now.

                  This is not a mature toolset yet by any means, but it does do it’s job very well, and I have several critical clients using it in production already. It has made my job building custom web applications and managing the databases that power them, much more efficient and even enjoyable at times. Still a lot more to do in order to make xPDO the end-user tool to trivialize custom database work in the CMS, but feel free to take a look at it and see if it might be worth adopting for a small test project. I’d be more than happy to assist you with understanding how it all works...