Beefy Boxes and Bandwidth Generously Provided by pair Networks Joe
laziness, impatience, and hubris

So, what *is* the best way to deliever dynamic content nowadays?

by Masem (Monsignor)
on Jan 04, 2002 at 19:42 UTC ( #136285=perlmeditation: print w/ replies, xml ) Need Help??

Part of my New Years resolution is to get moving on making changes to a site I operate; the old code was all done in 96, and are hacks upon hacks for features that were requested during the course by users; the database is certainly not the best and could probably deal with some normalization, and that was back before I use'd strict and -w all my code.

I want to 'moderize' everything, putting everything under a mod_perl/Apache engine to afford the speed bonuses, improve the database (currently in MySQL, but considering Postgres), and to make the site more 'personalized' for the end user by allowing them to select what styles or data they want to show, as well as any other means that I can think of. That's the goal, and I know that there are many ways to get there.

The problem is trying to determine the best way to get to that point given my requirements from where I am. I've got no qualms about rewriting my code to improve it, as there's not that much left to salvage. Fortunately, I'll be able to do this in parallel with the original site to avoid disruption in serving that one. Note that the hardware is a linux box on a 200mHz machine w/128Megs RAM; the site is active by endusers but not requiring massive amounts of bandwidth (maybe on order of 1000 unique hits/day), and the current site certainly doesn't strain the box to a great extent.

Here's some of the quandries that I'm thinking of:

  • To monolith-ize or not? I could have all of the dynamic content come from a single script that uses a module like CGI::Application, with a callout table for given actions, and such. This way, I also gain the benefit of having cookie/login handling in one place for the entire site as well as easy ways to add new actions to the system. However, the types of actions that the end user can do on the site are rather varied, but no more so than, say, on PM. Since I'm planning on using mod_perl, that means every server instance will probably load all of the available method codes that I provide, which can be a bit memory heavy. Alternatively, using several smaller apps with a common module for cookie/login handling would not task the memory, but would require more careful coding. (I'd probably still need to develop a common code module if I do use a monolithic app)
  • To XML-ize or not? I read this article on XML-based applications which described the use of XML as an intermediate layer between the DBI and the presentation, which allows for the application of XSLT style information before presentation. This can also mean one can develop XML streams ala the ones here at PM that the enduser can decide how to use to their likely. However, if the XML is only kept on the server side, these benefits can also be duplicated by judicious use of the various templating solutions like Template Toolkit 2, with the added bonus of reducing memory and CPU time before page generation. Since I can see that there are possible uses for a XML output from my site, I could either make the entire site use the intermediate XML layer, or I could simply provide XML when it's requested, possibly through a template solution again or just though the methods listed above, while the rest of the dynamic content is done directly through a template solution.
  • (not quite as perl-ish) MySQL or Postgres? I don't have to worry too much about locking tables or transactions or similar features as this isn't a ecommerce site. The only key thing is that with a properly normalized database, I may need to do subselect statements that I believe Postgres can provide but MySQL can't (though can be simulated with successive SQL calls). I know MySQL tends to be faster but is showing it's age, while Postgres provides many of the advanced SQL features in the professional grade DB system at the cost of speed. While my current DB is in MySQL, I don't believe I'm using many MySQL-specific features, so I'm not too worried about switching it over to a Postgres.
I'm sure there are other considerations that I've not yet thought of either, which I'd be happy to hear about. However with all of the current solutions that are out there, I thought it best to solicate the advice and experience of other monks before I broke new ground for the revised site.

"I can see my house from here!"

Comment on So, what *is* the best way to deliever dynamic content nowadays?
Re: So, what *is* the best way to deliever dynamic content nowadays?
by arhuman (Vicar) on Jan 04, 2002 at 20:32 UTC
    • To monolith-ize or not?

      It's a matter of taste, but I personnally like the (usual) idea of several scripts using common libraries.
      (database access, presentation, system interaction...)
      By the way I see no additional difficulty on managing cookie/session via several scripts instead of one.
      (did I miss something?)

    • To XML-ize or not?

      By making a clear distinction between the data and the presentation, XML is a "must have" nowadays,
      furthermore a lot of emerging technologies use it as input/output format or at least as a pivot format.
      One advantage I grant to XML over standard templating system, is that XML will provides you more tools
      (parser, checker, transformer...) than any templating system.
      It would be cool to use DTD to validate (for a particular browser) HTML code
      produced via XSLT translation from XML...
      The question is much "how to do it ?"
      The article gives an interesting way, axkit may be a good option too,
      anyway I'm sure some people here will probably give you the best advices on XML.


      Postgres! if your goal is to do the things the right way.
      Even if (beccause?) I've been using MySQL for 2 years at the office,
      I'd recomment MySQL only when speed is REALLY needed
      Foreign keys, trigger, views... make the DBA life so easier.

    "Only Bad Coders Code Badly In Perl" (OBC2BIP)
by mpeppler (Vicar) on Jan 04, 2002 at 21:53 UTC
    MySQL or Postgres...

    I'd use Postgres (well - actually I'd use the free Sybase release if this is on linux :-)


by sparkyichi (Deacon) on Jan 04, 2002 at 23:21 UTC
by lachoy (Parson) on Jan 04, 2002 at 23:37 UTC

    This sounds like an excellent fit for OpenInteract. (I'm a little biased, but still...) It provides much of the infrastructure you need for web applications -- authentication/authorization, security, centralized URL->action-mapping, easy data access (DBI, LDAP, ...), presentation framework using Template Toolkit, etc.

    Since you're using a relatively low-powered machine, it would probably be good to spend a few minutes looking at the common mod_perl usage of lightweight proxy servers sitting in front of heavyweight mod_perl servers. This way you don't have to start many mod_perl servers that eat up your memory.

    As for databases, PostgreSQL is more featureful than MySQL, easy to setup (as long as you're comfortable with ./configure; make; make install), updated often, low maintenance and well supported with not only DBD::Pg but also ODBC and JDBC drivers. I don't even use MySQL anymore unless someone specifically requests it.

    M-x auto-bs-mode

by perrin (Chancellor) on Jan 04, 2002 at 23:37 UTC
    I don't think there's much memory tradeoff between CGI::Application and separate modules. You should be preloading all of this in so it gets shared. Personally, I like to separate functionality into separate modules according to major site functions. For example, I might have a module for updating your user settings, and another one for browsing one of the data types published on the site. I also model the data objects as separate classes, so these modules I'm talking about are really just the "controller" part of a model-view-controller design.

    As for XML, I have found uses for it in data exchange but not in page generation. I just can't see the compelling argument for it. TT2 templates are easier to write than XSL. It's trivial to write a template that spits out an XML version of your data for a feed. Validation - well, TT will not compile an invalid template either. I can't see a good reason to use the much slower XML approach, except maybe to plump up your resume.

      Hi Perrin,

      Have you benchmarked recent XSLT solutions for this pre-conceived "slowness"? They are very very fast these days...

      As far as the advantages, I'll paraphrase Robin Berjon on this... The advantage he sees of using XSLT is that it's a very well thought out system, with a lot of history (DSSSL) behind it. With XSLT you aren't constrained by flat structures like many templating systems - and by that I include hierarchies where you have to "manually" iterate over the tree. With XSLT you just declare what bits of your data are meant to look like, and run it. But it's more than that... With XSLT you are not creating text. That is what every single other perl templating solution does - it takes some data and a template and generates text. This does not lead to efficient pipelining (which is a technique that most people in the perl world aren't really familiar with, because of this weakness, but a very useful technique nonetheless). With XSLT you take some a data structure, and a template which describes a new data structure, and you get a data structure at the end of it. Not text (and not really XML either - you have to think outside of that box :-)

      Oh, and one other benefit: multiple vendors.

      Matt (with a v.fat C.V.) ;-)

        I haven't benched the latest round of XSLTers, but the performance on XSLT has historically been pretty abysmal. Most of the time people (read "Cocoon") say they have solved the problem by caching the output, but not all output can be cached. I doubt that XSLT can be very fast when actually parsing XML (for the data and the style sheet), so presumably the biggest speed wins would come from caching the stylesheet as perl code, and from generating the data in a perl data structure that the XSLT processor understands instead of actual XML (thus skipping the parsing step). Are we there yet? And if we are, aren't we kind of re-inventing the wheel? Perl doesn't need XML to make generic data structures.

        Your point about not being constrained to a linear mode is actually one of the things I hold against XSLT, because all of the HTML monkeys I know like to think about page design in a linear mode. They don't want to specify a style for the product price; they want to write some HTML and say "the price goes here." It's just more intutitive to non-programmer types.

        I can see value in pipelining for working on data, but I would do all of that data mangling before I get to a page generation stage, so the template tool itself doesn't need to support it.

        Anyway, I have happilly used XML for other things and I don't usually take stabs at XSLT, but since he asked for opinions... I try to keep my preferences from coloring my templating guide too much (which now desperately needs an update, with new versions of AxKit, TT, Apache::ASP, etc. out).

Delivering Dynamic Content
by dbrunton (Novice) on Jan 05, 2002 at 07:23 UTC
    To monolith-ize or not?

    The short answer: yes. Some parts of our development want to be monolithic, and some don't. It's like Perl- some parts of Perl want to be OO, and some parts simply do not. mod_perl was a piece of this answer for our project (it's pretty monolithic) but it was too hard to make it spit out HTML all by itself. HTML::Mason pretty much solved that problem for us, and then we were left with the single remaining problem of how to represent the underlying data. Which is what your next two questions were about, so I'll stop at that for a minute.

    To XML-ize or not?

    This was an easy one for me, and I hope I can infect you with the zeal of the newly converted. If you can find a way to have your database do the things databases tend to be good at (e.g. indexing, search, etc.), and to keep XML around because it lets you have arbitrarily deeply nested data structures and other such niceties that Perl REALLY LIKES, this seems like an ideal solution to me.

    After playing with an early version of Kwin Kramer's XML::Comma, I liked it so much that I tracked him down and took a solemn oath to forever uphold the honor of XML::Comma and to never use PHP again. Or something. It really is insanely cool. Check out this post or just read my short summary:

    XML::Comma uses XML for two different purposes. First, it stores documents in untyped XML. Secondly, it uses a special flavor of XML to specify a single API for indexing (I'm using MySQL), storage (XML on the filesystem in my case), inverted indexing (MySQL again in my case), compression (GZip for me), Encryption (HMAC + symmetric key encryption here), and is capable of extending this interface to everything but washing your dirty undies.

    What this means for me is that when I use the Comma API to create a document a la:

    my $doc=XML::Comma::Doc->new(type=>'Foo'); $doc->name()->first('David'); $doc->name()->last('Brunton'); $doc->phone()->area_code('509'); $doc->phone()->num('5551212'); $doc->store('main');
    That last $doc->store() gives me (based upon my XML DocumentDefinition) a file on the filesystem that is gzipped, encrypted, HMAC'ed, sorted, and generally kept track of. It gives me corresponding fields in my database that are typed and indexed according to my specifications. It creates an inverted index for my search engine. And it does it many thousands of times every second even on my desktop machine, not to mention the quad Athlons we deploy on.
    I highly recommend checking it out. I did. But be careful. I started out just playing with it in my spare time, and now it's become my full time job to play with it ;)

    MySQL or Postgres?

    I use Comma with MySQL. We've done some benchmarking of Postgres, but haven't figured out a good reason (performance or features) to switch yet. Besides, I think XML::Comma is still Postgres/MySQL agnostic, so barring the use of any of the arcane indexing stuff, I think I could probably switch without much effort.

by markjugg (Curate) on Jan 06, 2002 at 02:39 UTC
    My two cents:

    I use CGI::Application in combination with Postgres and I'm very happy them both. I like organization that CGI::App encourages. In that framework, it's easy to create a single code line that powers multiple sites at the same time with different template sets and parameters. Because a maximum amount of code is in modules, potential re-use is maximized.

    I like Postgres because it's feature-rich. I've also used MySQL and found that it's often much faster to code a solution using Postgres. The one thing I hear in favor of MySQL is a "speed difference". In my real world application of using Postgres for dozens of projects for a website development firm, the speed has always been good. I'd throw mod_perl at a project before I would switch to MySQL. :)

    You can see an example of CGI::App and Postgres in action in my Cascade project, a content management system.


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://136285]
Approved by root
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
As of 2014-03-13 14:48 GMT
Find Nodes?
    Voting Booth?

    Have you used a cryptocurrency?

    Results (281 votes), past polls