Database abstraction, total OO refactoring, and more!!!

I am not familiar with Propel at all, but just from looking at its site and from talking to Jason, I am convinced that Jason is on the right track with his plans. Specifically I see a couple of critical problems that he’s solving:

Growing complexity: Whether we like it or not ModX is growing up and every new feature adds more complexity, interdependence, and potential for breakage in the core. The refactoring is the first step to untangling the spaghetti, but in the long term we need a much better object abstraction layer. I have a LOT of experience in this area (go look at http://www.enhyrda.org for an example of a Java based app server to which I contributed a lot of code) and it’s my sincere belief that without a good data abstraction layer, a project cannot grow. The whole discussion about the number of classes or number of DB tables is irrelevant since it’s all automagically generated and handled by the system. The developer never sees that stuff. Instead, the developer gets to model complex relationships and interdependencies without having to worry about SQL statements or hidden features in classes. The guy writing snippet code, never has to worry about how many generated layers of code is underneath the pretty facade that he sees. And best of all, as a platform, we’re no longer tied to MySQL as our database. Anyone who doesn’t think this is a good feature, please let me know so we can debate that

Speed of Development: The current learning curve for a new programmer in ModX is a little high right now. My belief is that a large part of that is due to tightly coupled architecture that requires a significant amount of digging before you can be productive in working with the core of ModX. By abstracting away the data layer of the application, and refactoring the existing code into easier to grok classes, I think we’ll see more developers migrate to ModX and in return we all benefit from that. In addition, this allows us a much easier way to have automated testing of code, which is critical to preventing functional and security errors.

I truly think that ModX is getting large and complex enough that we need a framework as comprehensive as Propel. Yes, the transition will require some learning and some work upfront. But from looking at the cost and benefits of the migration, I think it’s a definite win for the project.

Nice thoughts, Victor. I would surmise you’ll help us start writing some testing automation stuff, too?

Jason, will devs still be able to write directly to the database without breaking things horribly?

Quote from: rthrash at Dec 08, 2005, 03:45 PM

Nice thoughts, Victor. I would surmise you’ll help us start writing some testing automation stuff, too?

Uh no. I am just the ideas guy
Just kidding, of course I’ll help to contribute to unit tests as we get some of the new classes in place

Quote from: rthrash at Dec 08, 2005, 03:47 PM

Jason, will devs still be able to write directly to the database without breaking things horribly?

I can answer that -- A programmer can always grab a reference to the database connection and do what he wants. The idea of course is that hopefully they’ll never need to do that.

Okay, I’ve gotten wind of this a couple times and I’ve poked around the propel site a bit and here’s my two cents worth.

I think that propel gives us a few things that we really should have and it gives it to us fast. The fact that Jason could get all of this working so quickly is pretty cool. I’m all for a real OO api, I’m all for less duplication of code (having everything defined in one xml file instead of a db, "action," "processor" and API is pretty cool) and I’m all for supporting multiple databases (I’m not nearly the MySQL fan that a lot of people are).

Now here are my concerns:
Performance - from what I’ve seen not a big issue (I’ll sacrifice a little if it gives me a lot in return)
Simplicity - I have no problem with making the core complicated and using advanced OO and build methods but I do think we need to keep snippet creation simple. However, just because the API and the core is OO doesn’t mean that the snippets have to be OO. There’s a big difference between writing OO code and writing code that uses objects. So I don’t think that the snippets will have to be any more complicated than they are.
The Future - Before we get 100% commited to this I want to make sure that we’re not going to regret it down the road. So let’s do a lot of testing and watch the development of this and try to figure out where propel is going and how fast.

Jason, I think it would be very helpful to see an example snippet using Propel. Maybe a modification to an existing snippet so tha t we can see eaxctly what a propel version of a snippet would look like compared to a current version.

Quote from: aNoble at Dec 08, 2005, 04:10 PM

Now here are my concerns:
Performance - from what I’ve seen not a big issue (I’ll sacrifice a little if it gives me a lot in return)

Absolutely, but from tests so far I see no significant difference in either MySQL or PHP parsing time using the default content. More soon, as I complete more of the code refactoring to use the objects throughout the document.parser class.

Quote from: aNoble at Dec 08, 2005, 04:10 PM

Simplicity - I have no problem with making the core complicated and using advanced OO and build methods but I do think we need to keep snippet creation simple. However, just because the API and the core is OO doesn’t mean that the snippets have to be OO. There’s a big difference between writing OO code and writing code that uses objects. So I don’t think that the snippets will have to be any more complicated than they are.

IMO, once you understand the basic patterns of OO design, it becomes much easier to create and refactor code in the system. It’s like semantically correct XHTML: you can do a lot more automation and dynamic presentation using the semantic standards in XHTML, and the same holds true with OO patterns and code design/maintenance/automation. So once you get over the initial fear of change, which comes any time you shift development paradigms, I think you will all find it simpler and more intuitive, though it will require more collaboration. But I don’t thin that’s a bad thing for the project anyway.

Quote from: aNoble at Dec 08, 2005, 04:10 PM

The Future - Before we get 100% commited to this I want to make sure that we’re not going to regret it down the road. So let’s do a lot of testing and watch the development of this and try to figure out where propel is going and how fast.

Absolutely, I’ve committed us to nothing; just wanted to make everyone aware of the experiment. But the more I think about it, even if we decide to come up with our own solution to this, there is absolutely no reason we can’t start with this open source code. Why re-invent the wheel from scratch when you can refactor and re-engineer a wheel someone has already provided you with? We did this in several commercial projects I’ve been involved with, complete changing and customizing the open code to fit our exact needs.

The only complication in my eyes, will be setting up your development environment, for which I hope we can work together to quickly document the steps required for Windows, Linux, and Mac developers to start working with Propel, Creole, and Phing; I got it setup with my existing Eclipse/PHPEclipse dev environment in about an hour tops; you’ll be able to do it in a matter of minutes with the instructions, if you already use Eclipse. And unfortunately, the only cross-platform solution that makes any sense for this is Eclipse with PHPEclipse extensions; and you can use the Ant build tools in Eclipse to edit, control, and run the Phing build scripts, add SVN integration, and, well, you get the point. It’s the best overall development environment for the project IMO, regardless if we choose Propel or not.

Quote from: aNoble at Dec 08, 2005, 04:10 PM

Jason, I think it would be very helpful to see an example snippet using Propel. Maybe a modification to an existing snippet so tha t we can see eaxctly what a propel version of a snippet would look like compared to a current version.

Absolutely! I will most likely provide all of the snippets in the current core codebase, including the ones in the default content install, refactored along with the core when I present this.

Very on topic: http://www.mysql.com/news-and-events/web-seminars/mysql-nasa-zend.php

My views remains the same.

While Propel has a lot to offer in terms of OO database the fact still remains that we are heading into something that could either break us or make us.

On my list of concern is:

1) Complexity - What is requires to develope code using Propel? I see where jason has pointed that there will be docs to help users setup phing, propel and creaole. This concerns me very much. How easy will it be to develope and package apps in MODx should we go the propel road? Is this a point an click experience? If not then that would dent the dream of a Visual Experience? Another thing comes to mind is access levels. Would a developer need special access to his/her server to run these third party tools?

2) Performace - Propel is NOT a database abstraction framework, right? It’s an OR mapping framework. Propel uses Creole for DB abstraction, right? My question then what are we pursuing? DB abstraction or OO database?
Propel’s OO layer will most likely be used inside the core APIs which IMO adds processing overhead to these APIs. The performance will not be the same as if we were using straight creole/dbapi or standard SQL syntax, correct?

3) Scalability - How well does propel scale when we start talking about hundreds of tables? Sure MODx might not use so many tables but what if someone was to use MODx to create such an application? How well will such an app function under MODx? How many request/second will such a website be able to handle?

These are my concerns if these can’t be addressed then I think we are heading in the wrong direction.

The Visual Dream is to enable developers to use the MODx interface to do everything from building and packaging web sites and web apps to deploying such packages. Developers should never have to use the command prompt ever.

Raymond,

Some good points. I’ll address a few from my view point:

Quote from: xwisdom at Dec 09, 2005, 12:03 PM

My views remains the same.

While Propel has a lot to offer in terms of OO database the fact still remains that we are heading into something that could either break us or make us.

It’s not quite that drastic. This is just an addition to make an already good project even better. I don’t think this will "break us", rather it might help us. If it doesn’t -- then we move on.

On my list of concern is:

1) Complexity - What is requires to develope code using Propel? I see where jason has pointed that there will be docs to help users setup phing, propel and creaole. This concerns me very much. How easy will it be to develope and package apps in MODx should we go the propel road? Is this a point an click experience? If not then that would dent the dream of a Visual Experience? Another thing comes to mind is access levels. Would a developer need special access to his/her server to run these third party tools?

Let’s make sure that we’re very clear about what the OR framework is going to do. It’s not generally going to change how the average web site developer uses ModX. I think the goal is that snippet writing stays mostly the same (and certainly not more complex), with some minor changes to account for the new underlying framework. Where the real value is (at least to me) of this project is, is that writing plugins and major modules becomes much, much easier. It’ll also be easier to work on ModX internals for new developers. Now if a snippet developers wants to take advantage of a new, more powerful, data model, then yes it’ll different from how it’s done today but certainly not more drastically complex.

2) Performace - Propel is NOT a database abstraction framework, right? It’s an OR mapping framework. Propel uses Creole for DB abstraction, right? My question then what are we pursuing? DB abstraction or OO database?
Propel’s OO layer will most likely be used inside the core APIs which IMO adds processing overhead to these APIs. The performance will not be the same as if we were using straight creole/dbapi or standard SQL syntax, correct?

Performance is a red herring here -- sorry, but it is. We spend so many more cycles in processing/accessing data than we do in any kind of object creation/manipulation that adding an abstraction layer will not introduce a performance hit that will be noticeable to anyone. And if there is 1% (or whatever) hit in performance, in my opinion, it’s worth it to have it get all of the SQL code that’s currently intermingled with business-level logic, and creating a true multi-tier architecture that makes it possible to write code that’s not fragile or database-dependent.

3) Scalability - How well does propel scale when we start talking about hundreds of tables? Sure MODx might not use so many tables but what if someone was to use MODx to create such an application? How well will such an app function under MODx? How many request/second will such a website be able to handle?

Multiple tables don’t cause MySQL to slow down. It’s a crappy DB, but it’s not that crappy But the performance concern is a valid one. And I don’t think we can know the answer to it until the framework is built. I do know that if someone is running a commercial site that’s concerned about that much scalability and performance though, chances are they aren’t running MySQL. And this is where the DB abstraction layer helps us again -- we can port to Oracle or any other commercial DB much easier with these changes.

These are my concerns if these can’t be addressed then I think we are heading in the wrong direction.

The Visual Dream is to enable developers to use the MODx interface to do everything from building and packaging web sites and web apps to deploying such packages. Developers should never have to use the command prompt ever.

I agree 100%, and there’s no reason why this new framework should change that.