Maybe we should release 0.9.7 instead

Raymond’s parser has some really great ideas going on in it. Unfortunately, it’s not quite ready for prime time. I think we can not only address the issues that surfaced with it, but also take it a few steps farther. Raymond worked magic, especially given the Eto-legacy. However, to take it to the next level in an accelerated fashion, I’m now convinced we need to depart from the current core architecture. You’ll soon see why and I hope you agree.

I realize we can’t alienate our current users and existing sites. So what I did was to challenge Jason to dust off his 1.0 work and get it working in the PHP4 world (and PHP5.1.x too of course), but also using the current manager and installer, and some "real world" content beyond the basic default content. The biggest problem with Jason’s stuff is that no one outside of him has really seen it, and quite frankly when he describes it, it seem way too complex and intimidating for me -- I call it too "wooshy".

After the last two days of work, I think it’s pretty much there and then some. Really, I can barely contain myself as this is so close to where I was hoping we’d be sometime mid-next year! As a proof of concept, I took our family’s business site (http://corganics.com/) which is chock full of Wayfinder and Ditto calls and is also running a development version of ShopX, which means custom plugins and modules (test bed at http://bycarter.com/).

So, what does back-porting all the 1.0 stuff get us? A "few" things that one can make a good argument for skipping the 0.9.5 release entirely, like:

[*] revised call syntax and recursive capabilities using Raymond’s syntax and not using regex for optimal performance
[*] conditional MODx tag calls (like :IF and so on) easily accomplished via nested snippets
[*] ability to pass parameters into chunks
[*] probably 95% compatible with current code ... and even then, probably closer to 99% of existing sites should work with a bit more external testing and feedback -- changes needed to be compatible are quite simple (mostly changes to snippets to no longer use internal API methods like Newslisting using $modx->mergePlaceholderContent($modx->mergeChunkContent($templateContent)) which works slightly differently since there is no longer a forced execution order...)
[*] speed comparable to today (even during emulation of the current 0.9.2.1 API!)
[*] no EVALs anywhere (except for the @EVAL @BINDING)
[*] db-based sessions that can keep you logged in forever if you like with no tricks employed
[*] ability to have dynamically generated snippet names in addition to dynamic parameters
[*] a robust and proper caching system that will become even more fast over time (making it considerably faster than today)
[*] "alias" or "symlink" documents -- an extension of weblinks that are references/pointers to other documents but that hold your place in menu systems (this makes my inner-Wayfinder SO happy )
[*] correct header responses, including proper 200 (redirect), 404 (not found) and 503 (network unavailable/maintenance) responses that combined with the caching system and new parser I suspect will forever banish the sporadic "mystery blank pages" to the past
[*] true Object Oriented and consistent API throughout the core
[*] mostly documented API (should be completely so by the public/official release)
[*] once we have a new manager, the ability to have MODx run on virtually any database with a small amount of development work on PHP 4.3.x+ including optimized performance in PHP 5.1.x+ (via native PDO)
[*] ability to use the MODx class outside of MODx, with no separate API needed
[*] a real web services API and a real Ajax services API (with a bit more work)
[*] easily extend or override any part of MODx without hacking any core code whatsoever
[*] any string of content you can pull from anywhere can be run through the parser, just as if it were actual MODx managed content (this means you could put MODx tags in Wordpress and write a Wordpress plugin to parse it through MODx!)
[*] simple APIs to virtually any data structure/third-party system like Wordpress, Joomla! or Textpattern (MDPro done already because Jason needed it )
[*] really simple ability to create content from the front end including full TV support

Jason promises a commit of this this week (in a new branch), along with a downloadable demo install. Regardless, we still need to back out the stuff that’s breaking the trunk and integrate all the great commits that have been made this week, so that will happen too... but folks... this is REALLY exciting stuff: I can work exactly how I’m familiar with working now, but I gain SO much more capability when I’m ready.

I’d like to see us getting the trunk working again with all the fixes/commits from the last week or so (great work folks ) and release a public beta test to see if it solves some problems for the users. Then we could internally test Jason’s work on a broader assortment of real-world content. Making the assumption it goes as well as it did for corganics.com, I’d then like to see ASAP before the end of the month a public beta with all the manager fixes integrated and hopefully the much-cleaned-up skin that Jeff’s been working on.

Thoughts?

Agree 100%!

Thoughts ??

I almost had a brain collapse reading your post I have no thought at the moment except you guys are just amazing
What I did was check we weren’t the 1st of April, which of course I knew we weren’t but my, this is e-x-c-i-t-i-n-g !

I was nowhere near suspecting this kind of breakthrough !!!
Wooshhh, really I am so glad I jumped on board almost a year ago

I’ll have to let the info sink in before I have thoughts again
Now if you’ll excuse me I have to read that again to make sure I am awake !

Edit : Forgot to say : I am all for it, of course !
This is such a daring move, we have to make some extensive testing... I am pretty much excited at the thought ! I have a least four live sites I could mirror to do some testing

After re-reading your post, I am not sure I get the timeframe entirely :
- we do release an interim beta with current 0.9.5 improvements without the new parser, this week
- we work on 0.9.7 this month and at the end of september, offer a public beta of the new baby

Did I get it right ?

I’ll happily put my demo site and template showcase site on this, as well as testing it locally of course. And a couple of people I sent links to the Corganics site were very impressed with it.

Love it all apart from "instead".

I do think we should be prepared to take 0.9.5 out of beta if it proves to be more solid and a better starting point than 0.9.2.1 (which I think it will, once the parser is reversed out). The change from 0.9.2.1 to "current trunk -parser" is largely a bug-fixing / feature-adding exercise, whereas the proposed 0.9.5 to 0.9.7 leap is quite a radical change. Thus 0.9.7 will need a longer public beta period than 0.9.5.

Existing users should be warned that the time gap between 0.9.5 and 0.9.7 is expected to be short, but that doesn’t mean that newcomers should still be installing 0.9.2.1 in early October, nor should it mean that those investigating the demo on OpenCMS should still be looking at that months-old version.

In case it’s not clear, the main principle of reverting 0.9.5 to pre-parser, re-adding all the other changes, and then releasing that in whatever form before switching focus to the exciting 0.9.7 release... this all meets with much approval and excitement here.

Simple APIs to virtually any data structure/third-party system like Wordpress, Joomla! or Textpattern (MDPro done already because Jason needed it Tongue)

Just so I’m clear; there’s a mechanism for creating such APIs *and* there will be some included with 0.9.7? I hereby volunteer to sort out the osCommerce and phpBB APIs.

I would normally agree with you Paul if it weren’t for a few bugs that I think will be impossible or nearly so to track down that NEED to be fixed for the next release regardless of the version number.

In fact I submit that based on my experience with the code being ripped out of corganics.com and having it work in the new system – coupled with the actual amount of time that went into Jason’s release in reality (more like the culmination of the last year’s worth of work actually) – it’ll be faster to have a better system in less time that addresses all the bugs vs. deferring things like the mysterious blank page bug in particular. That’s a very real bug/mystery that’s virtually impossible to reproduce without access to specific testing environments, and we’ve banged our head on it for far too long. I’ve seen it happen on cheap shared host; I’ve seen it happen with dedicated servers. It’s a weird configuration corner-case I suspect. There’s also the really weird case of TVs not being updated and mysteriously duplicating at times that I hope/suspect goes away.

The current core is so procedurally twisty in so many directions it makes it quite challenging to figure out what’s going on. Add in that the header response issues go away, and several other things, and I’m leaning towards it being much faster to ship Jason’s core than a bugfix version. The bulk of the changes in the tracker are manager related and apply equally to either version.

Now if you’re suggesting we back off the roadmap/slated bugs and release that as 0.9.5 ... eh ... not quite so sure what I think of that.

Quote from: PaulGregory at Sep 06, 2006, 04:31 AM

Love it all apart from "instead"

...

In case it’s not clear, the main principle of reverting 0.9.5 to pre-parser, re-adding all the other changes, and then releasing that in whatever form before switching focus to the exciting 0.9.7 release... this all meets with much approval and excitement here.

Absolutely agreed Paul. I just discussed this with Ryan some more on the phone, and I think we’re both all for getting the current snapshot of MODx, minus Raymond’s parser commit(s), and with a few more merges from recent bug fixing activity in several contributor branches, out the door as soon as possible. I’d like to think we could get this done by early next week.

Quote from: PaulGregory at Sep 06, 2006, 04:31 AM

Simple APIs to virtually any data structure/third-party system like Wordpress, Joomla! or Textpattern (MDPro done already because Jason needed it Tongue)

Just so I’m clear; there’s a mechanism for creating such APIs *and* there will be some included with 0.9.7? I hereby volunteer to sort out the osCommerce and phpBB APIs.

This mechanism is called xPDO, and it’s really available now, though I am certainly still working on and improving it. xPDO can reverse-engineer just about any MySQL database structure into an XML schema and then turn around and generate all of the code needed (a class file and a metadata map file) for all basic CRUD operations, all in about 1 second total. And you can easily define aggregate or composite relationships between tables in the XML schema, to gain further functionality between related objects, like cascading delete emulation for composite object relationships, or cascading saves with any related objects. I’ve used xPDO so far to do the following kinds of projects in the context of MODx:

* Build custom web applications from scratch and skip all the CRUD coding; just add the functions to the generated classes to define any custom business logic.
* Easily author scripts to import data from one system to another (e.g. PostCalendar to a custom calendar database, MD-Pro news stories to MODx documents, WP posts and comments to MODx documents, etc.)
* Easily author snippets and any other PHP components using the API’s directly (i.e. show content from WP or PostNuke directly in MODx with no import)
* Consistent CRUD API’s for any data you want to access, that take full advantage of native PDO where available, or runs using a PHP 4-compatible PDO/PDO driver emulation that’s included with xPDO for MySQL, SQLite, and PostgreSQL.
* Reverse-engineer and migrate all current MODx business logic/functionality into these generated classes; makes it super easy to convert your existing code to an object-oriented paradigm, that works in PHP 4 as well as takes advantage of the new features of PHP 5.

So soon, you’ll be able to do this with any database supported by a native PDO driver. Just a little more work to go on xPDO for that. And the new main class for the MODx Ryan is talking about here is actually implemented as an extension of the xPDO class itself (which extends PDO). So all of the functionality of PDO and xPDO is part of this core MODx class as well.

Ryan - Well, that real bug is in 0.9.2.1. Either we have newbies downloading 0.9.2.1 or 0.9.5. Either way, the bug is there.
Having considered this some more, I think the best solution is to have the public beta of 0.9.5 on the official download page.
(I would prefer to release now as 0.9.4, just to indicate that it’s not quite the wondrous 0.9.5 of legend but better than 0.9.2.1, if that wasn’t going to be major league awkward).

Jason - I’ve seen xPDO, I must admit I’d backed away from playing with it when I heard you were having problems with PHP 4.

Quote from: OpenGeek at Sep 06, 2006, 08:53 AM

xPDO can reverse-engineer just about any MySQL database structure into an XML schema and then turn around and generate all of the code needed (a class file and a metadata map file) for all basic CRUD operations, all in about 1 second total.

And you can easily define aggregate or composite relationships between tables in the XML schema, to gain further functionality between related objects. [...] Consistent CRUD APIs for any data you want to access, that take full advantage of native PDO where available, or runs using a PHP 4-compatible PDO/PDO driver emulation that’s included with xPDO for MySQL, SQLite, and PostgreSQL. [...]

So soon, you’ll be able to do this with any database supported by a native PDO driver.

I really want to nail down what is being said here. I’m assuming that "database" in the last sentence I quoted there means the db format, ie MySQL / SQLite, rather than the specific database for an application ie phpBB / osCommerce.
For the sake of argument, lets say everyone loves "Hypothetical" an xAMP App for pizza recipes and we want to create an API for it that we can use in MODx.

• xPDO can generate a class and metadata map just by giving it the MyHypoDB MySQL db name and appropriate user details in 1 second flat.
• A human being can then document this.
• A human being can then create a class which connects together tables, such that asking for the creator of a pizza returns the username rather than the user ID, and asking for the ingredients returns a list from multiple table. Or that creating a pizza also increments the "Pizzas Created By User" count by 1.
• A human being can then document these extra classes.
• This API can then be used by a MODx snippet which displays the most recently added pizza.
• This API can be shared with other people.

Therefore, although xPDO can generate a basic API, it’s not in and of itself much more useful than the existing "connect to db, get this field" mechanism... until a human comes along and adds the logic, documentation and example snippet.

And *that* is what has been done with the MDPro API?

Given that the API is written for xPDO, not MODx, where would the primary place be for the hosting and sharing of these APIs?

Finally, can we call an xPDO API an xAPI and pronounce it Zappy?

Quote from: PaulGregory at Sep 06, 2006, 09:55 AM

I really want to nail down what is being said here. I’m assuming that "database" in the last sentence I quoted there means the db format, ie MySQL / SQLite, rather than the specific database for an application ie phpBB / osCommerce.

Yes, I meant database platform (i.e. MySQL, SQLite, PostgreSQL, etc.) in that context.

Quote from: PaulGregory at Sep 06, 2006, 09:55 AM

For the sake of argument, lets say everyone loves "Hypothetical" an xAMP App for pizza recipes and we want to create an API for it that we can use in MODx.

• xPDO can generate a class and metadata map just by giving it the MyHypoDB MySQL db name and appropriate user details in 1 second flat.
• A human being can then document this.
• A human being can then create a class which connects together tables, such that asking for the creator of a pizza returns the username rather than the user ID, and asking for the ingredients returns a list from multiple table. Or that creating a pizza also increments the "Pizzas Created By User" count by 1.
• A human being can then document these extra classes.
• This API can then be used by a MODx snippet which displays the most recently added pizza.
• This API can be shared with other people.

Therefore, although xPDO can generate a basic API, it’s not in and of itself much more useful than the existing "connect to db, get this field" mechanism... until a human comes along and adds the logic, documentation and example snippet.

Well, instead of a human being defining classes that connect tables together, a human being defines the foreign key relationships between two tables/objects, or an intersection class (for one-to-many and many-to-many relationships) with these relationships; this is done in the XML schema that is produced from the reverse-engineering task. You add the aggregate and composite definitions, generate the classes and meta data maps (or just regenerate the maps), and you have instant access to all related objects...
i.e.

or

The only time additional classes beyond the generated ones are needed is when dealing with some kind of complex business logic that involves several persistent classes generated by xPDO. And the only additional documentation needed will be when you add or override a method to one of the classes; but this is rarely needed from my experience with using it so far. I mean, if you want to develop an API to completely replace an applications normal API, it would take some in-depth work, as I have invested in the modeling of the MODx application, but this is rarely the purpose for using it. There are a lot of features already built-in to xPDO API’s which your generated model will instantly be able to take advantage of.

Quote from: PaulGregory at Sep 06, 2006, 09:55 AM

And *that* is what has been done with the MDPro API?

Not exactly, I just reverse-engineered and forward-engineered the MDPro DB, defined only the relationships I was interested in, and used the generated classes to simply interact with the MDPro data directly.

In another instance, I took the DB from a commercial Ajax/Flash gallery management system and within an hour was able to integrate the user system (one way sync from MODx to the app) and create snippets to display random images that are part of the Flash gallery.

Quote from: PaulGregory at Sep 06, 2006, 09:55 AM

Given that the API is written for xPDO, not MODx, where would the primary place be for the hosting and sharing of these APIs?

xPDO in my view, is just a tool for kickstarting the development of a custom application that requires persistent data. In the case of MODx, I reverse engineered the current DB, defined all the relationships, added some inheritence to start converging the user systems and working toward the modResource/modElement model, and simply migrated the appropriate legacy functionality to the appropriate classes for handling that. Sharing other API’s built with it should be no different than sharing some other kind of add-on for MODx.

Quote from: PaulGregory at Sep 06, 2006, 09:55 AM

Finally, can we call an xPDO API an xAPI and pronounce it Zappy?

Hey, I kind of like that...

Quote from: OpenGeek at Sep 06, 2006, 11:05 AM

I mean, if you want to develop an API to completely replace an applications normal API, it would take some in-depth work, as I have invested in the modeling of the MODx application, but this is rarely the purpose for using it. There are a lot of features already built-in to xPDO API’s which your generated model will instantly be able to take advantage of.

I think we have to be very careful with the marketing and terminology here. To me, an API with just CRUD abilities but no application-specific functions isn’t really an API at all. To me, API infers application, not dataset. User expectations need to be very carefully managed. Yes, saying "Simple API" or "CRUD API" can be seen to lower expectations and qualify the claim but there will still be people who expect it to be more application-function-y. Personally, as much as I like to say xAPI, I’d strongly suggest we drop the A and P and just call it an Interface.

It’s potentially dangerous to provide a simple way of creating, updating and deleting without it being clear to coders that there’s no more logic present than if they just used MySQL directly - eg deleting a user with the basic xAPI will orphan lots of recipes, whereas deleting with the Hypothetical API would move them all to Guest.

That said, what *does* excite me *is* completely replacing an application’s normal API. Or in some cases, doing what an application’s API would do if it had one!