Database abstraction?

Do we have any sort of consensus on the DB abstraction?

After spending some time thinking about it, I favor the EZSql method, but I'm not sure where that conversation currently stands.

Depending on how we decide to proceed, I will go ahead and volunteer to redo all of the core code to use the new DB abstraction, if desired.

I believe the great performance of Etomite is because of the missing database abstraction layer. By creating providers for each template object, that is plenty of abstraction for the general site admin, and everyone can optimize their providers for their particular repository, be it a database or not. I think the idea of the Template Object core is to actually abstract the storage of information to any provider, so database abstraction does not address the entire problem. I think if we choose to go with something like the Template Object idea, database abstraction should be left out altogether.

I don't follow that logic at all. Just because we abstract objects, we should lock someone into mysql? What about those who want to use postgres or MS SQL or flat files with SQLite or ... ?

What am I missing here?

I don't follow that logic at all. Just because we abstract objects, we should lock someone into mysql? What about those who want to use postgres or MS SQL or flat files with SQLite or ... ?

What am I missing here?

no, the idea of pluggable providers for each TVO if you will, abstracts the access method to any database, file, or otherwise, by allowing you to write code directly for that platform -- this gives you the benefit of allowing each accessor to have optimized code for it's target. For a specific database platform, for a specific xml parsing tool, or a basic text file reader/writer.

in other words, you could use EZsql if you wanted to, though you would add an additional layer, or you could download the Oracle 9i provider from the library for access to data over here, and the XML provider for data over there

then, by adding federation capabilities to the providers, you can aggregate say an existing custom web user database with a brand new LDAP repository your business partner is using, simply by defining multiple providers for your object

making any more sense?

Sorry, not yet, no-- since all of the internal API calls have to call *something*, what are they calling out of the box?

Right now we have hard coded references to mysql in snippets, in the core, in the mod, basically sprinkled all over the place.

It seems to me that if we are going to provide something easy to use straight of the box but also powerful enough for others to come along and build upon we are going to have to provide a low level API for system access-- both to the file system and to the datastores.

We cannot, for instance, have snippet/mod/information providers just coding against system calls, they need to be coding against a common API. That way if someone comes to the site and sees a feature they like, Average Joe user doesn't have to understand that that particular guestbook/TVO/what have you requires Postgres to run.

Am I making any sense? Sure, we should have a wide range of data backends for people to pick and choose from, but all of them should be required to support a standard API that we define. Otherwise we aren't any better off (at least in my mind) than we are right now. In fact, I'd say we are worse.

A lot of this comes back down to what are we: if we want to have a nice package that Average User can come look at, play around with, download, and use, then we are going to have to be able to hide the complexities of "download an Oracle 9i provider from there or an XML provider from over there" and ensure that while we do support all of those things that we don't have trade offs that the XML can do this and the Oracle 9i can do that.

To me having a standard in core API that can be used consistently across the datastore just makes sense. Someone developing a TV or using the system or even making new snippets shouldn't have to know anything about how the data is being stored on the system -- theirs or anyone elses.

Am I making any sense here?

and I think my idea is the same as yours, just allows more flexibility.

To help reconcile our perspectives, let's focus on the layers as they would be in my model...

layer 1: documents, chunks, templates, snippets, TVO's etc.
this is the layer for users to administrate and where ease of use is absolutely required -- between the existing Chunks, Snippets, TV's, Placeholders, and the existing aresenal of symbolic template substitions we have available, I think this is in good shape -- accessing what I'm talking about is invisible at this layer, other than when building TVO's (e.g. defining your object data providers, which could also be used internally to abstract the storage of etomite data)

layer 2: API
at this level, we are still abstrated from the data providers; all that matters here is interacting with the existing API (or whatever else we come up with to add to it); on the back-end, the API would simply make calls to the object abstraction layer to decide what access method to use in gathering the data, in the same way that TVO's would, since they are basically template level representations of API calls

layer 3: data access providers
here is the magic layer, where you can simply define where, how, etc. all data is going to be stored, retreived, and managed, through the administration interface; at this level there would be a provider to the actual storage repository that is the direct access code. This then allows us to author providers that use EZsql, or providers written directly to Oracle, or providers to an xml parser, etc. this does add a layer, but I think we can find a way to efficiently get from API to provider

if we can find the approach to this that doesn't detriment our incredible performance, I really think we have a winning idea here

I'll let this issue rest and let others digest it before continuing on with it -- it's obvious a big decision to make, but I'd like to conside this approach at least; the idea is getting clearer and more mature based on other discussions in these forums as well

I would also want to suggest learn some technique from Eclipse, HTML-Kit and Smarty.

Utilize plugins as a way to extent the core.

The plugins could be either from a files in a folder call "plugins" or a snippet in a database table.

It will make the CMS as simple as one wish or as complex and powerful as one desire to make it.

The real important point is a choice and the flexibility of the CMS.

I tell ya many people come to our site asking for a CMS of this kind and when I wrote an article on "My dream CMS" it was a hot topics. The plugins was a common theme.

I hope Modx or whatever it will be call can be those people's "Dream CMS"!

i think the oss cms community would eagerly embrace modx if its data access were built upon an object relational mapping (ORM) provider, like propel. drawing from the success of the j2ee world, projects built in this fashion tend to gain a lot of momentum, because they play nicely with each other. we can learn a lot from the successes of Hibernate, and the related projects that use it as their data abstraction core.

if modx manages its data through an ORM provider, the possibilities are truly limitless:

1) we realize jason coward's requirement of layer 1/2/3 separation. a good ORM already has these layers abstracted for us.

2) modx can be used as it is today, as a great foundation on which to build content-managed sites from scratch.

3) modx can be dropped into an existing application with an ORM (propel) backend for instant CRUD.

4) we "know" the relationships between all of the content in our system because it's defined in the ORM layer.

5) developers can implement higher-performance, more scalable data models for their underlying content -- we would not be limited to content, TV, content_TV relationships. any data backend, so long as it's expressed in the ORM layer, would instantly integrate.

there's a number of us at my company willing to put some effort into this. i think it could make the difference between YACMS, and an emerging cms standard that draws energy from other active projects.

phew .... I'm not anywhere that deep into code but I must say that that really does sound intreging and really would like to read/see more of what you're talking about.

It seems to me that if we are going to provide something easy to use straight of the box but also powerful enough for others to come along and build upon we are going to have to provide a low level API for system access-- both to the file system and to the datastores.

We cannot, for instance, have snippet/mod/information providers just coding against system calls, they need to be coding against a common API. That way if someone comes to the site and sees a feature they like, Average Joe user doesn't have to understand that that particular guestbook/TVO/what have you requires Postgres to run.

I would have to agree with infoclipper. Any DB API that we use must allow developers to focus on their apps and not on the incompatibilies between different versions of SQL databases. In other words a developer should not have to worry about writting SQL that's compatible with MySQL, MSSQL, etc.

IMO something like a function call for example, dbGetDocuments($fields,$joins,$filters,$where,$limit) would return a dataset of records. Now the $filter, joins and $where clauses would be based on a very simple syntax that's compatible to the supported SQL databases.

As is now we all write to MySQL. And everything works out of the box.

At times we have to use a mixture of SQL and php to accomplish some workarounds due to incompatibilities with different versions of MySQL. The question now is how would such incompatiblies be addressed with a DB API? How would perfomance be affected?

I would suggest that post a working demo of how these things can be done. The key thing to remember is that such solutions should be easy to implement and should have a low learning curve.