A cachefu with less monkeypatching
==================================

Current patching in cachefu
---------------------------

Currently, cachefu does a lot of monkeypatching. This is a list of the
files in which monkeypatching takes place:

CacheSetup/patch.py -
  OFS.Image/File are patched so that they're associated with the
  PolicyHTTPCacheManager to get control over old-style files and
  images.

  A ``.modified()`` method is patched into old-style files and images
  and into filesystem files and images to support asking them about a
  last modification date.

  The catalog is patched: ``moveObjectsByDelta()``,
  ``uncatalog_object()`` and ``catalog_object()``. All now call a
  method that purges squid (except for uncatalog_object) so that the
  object in squid gets purged whenever it changes. "Changes" in this
  case is defined as "when the catalog changes". It was a good way to
  detect it for older plones, but at the moment not everything is
  detected this way. IObjectModified event to the resque?

  The squid purge method also increments a counter (also for
  uncatalog_object) inside the cache tool: this is used as a sort
  last-modified-date for the catalog. This counter is used quite a lot
  by cachefu to keep everything updated, especially for logged in
  users.

  ResourceRegistries' ``cookResources()`` is patched so that
  abovementioned catalog counter gets incremented everything something
  changes inside ResourceRegistries.

  ``manage_changePermissions()``, ``manage_permission()``,
  ``manage_acquiredPermissions()`` and ``manage_role()`` are patched
  to increment a counter every time the relationship between
  permissions and roles change. This counter can then be used to
  determine freshness.

CacheSetup/patch_cmf.py - 
  ``FSPageTemplate`` gets patched so that the filesystem (=skin
  directory) template doesn't get rendered if a "304 not modified" can
  be returned. Rendering the template is expensive and it isn't needed
  if a 304 can be returned. Secondly, cache headers are set after the
  page template has been rendered.

  The pagetemplate rendering mechanism, ``TALInterpreter``, is patched
  likewise. 304 interruption and cache heading setting.

CacheSetup/patch_interpreter.py -
  This is a patch to enable macro caching. It looks like it only works
  with plone 2.5 (not 3.0) at the moment because of the import of the
  TAL.TALInterpreter, which isn't available anymore in plone 3.0's
  zope version.

  Many of the people at the sprint didn't seem to like macro caching,
  so it might be best to just ignore it for the time being. In that
  case the patch can stay, but will only be useful for 2.5.
  
  Note: We've indeed removed it in trunk (i.e. for 3.0) on the PIKtipi 
  Sprint (tomster, fschulze)

CacheSetup/patch_utils.py -
  Actually, this is just a utility class that provides some handy
  methods for patching files.

CMFSquidTool/patch.py -
  Just a utility class for patching. What's the possibility for
  merging it with above one?

CMFSquidTool/queue.py -
  ``reindexObject()`` and ``unindexObject()`` of CMFCatalogAware and
  CatalogMultiplex are patched to put objects in the squid purge
  queue. Didn't ``CacheSetup/patch.py`` already to that? Might be a
  chance for some serious code cleanup here.

CMFSquidTool/threadinglocal.py -
  No cachefu-specific patching. It is basically a copy of 2.4.3
  ``threading.local`` library. Didn't plone 2.5 basically require a
  python 2.4? In that case, this module might be entirely
  unnecessary. Ah, no. Plone 2.5 runs with zope 2.8 also, which runs
  on 2.3.5.


More research: response headers and pagecache storing
-----------------------------------------------------

Ehm, what about setting the response headers for something else than
templates? And grabbing a nice fresh rendered content object for the
pagecache? I expected that to need some patching, too. Haven't seen it
till now, so more grepping is in order, especially for ``response``,
as that is where headers get set.

CacheSetup/cmf_utils.py -
  ``_setCacheHeaders()`` sets the cache headers according to cachefu's
  wishes. ``_checkConditionalGET()`` checks if a 304 should be send.

  This file doesn't do any patching, though. It gets used by
  ``patch_cmf.py``.

PageCacheManager/PageCache.py -
  ``ZCache_get()`` and ``ZCache_set()`` retrieve or store pagecache
  items and set the headers (``X-pagecache``) accordingly. Hm, I
  almost forgot: items can be associated with cache managers. That's
  another angle to explore. Apparently cache managers automatically
  tie in at the start and finish of the process when associated with
  the content object that is being served.

  Looking at a wiki page where they discussed zope3's cache manager
  revealed that Tres Seaver had a strong preference for completely
  separating the caching of results (like in a cache manager) and the
  setting of headers. Those ought to be different products. Jim Fulton
  agreed. I also tend to agree :-)

  An almost logical result of this thinking is to look at the
  replacement cache managers offered by zope3 instead of at the
  existing zope2 cache managers as used nowadays by cachefu.

  Lovely systems made a viewcache_ for zope3 during the 2007
  snowsprint. That one caches zope3 views. What is left to do is a
  cache for regular skin templates (pagecache could handle that for
  the time being, perhaps) and for python scripts. Well, python
  scripts are something separate: separate RAMcaches per script as per
  the instructions in the "speeding up zope" talk at the 2006 plone
  conf.

.. _viewcache: http://www.lovelysystems.com/batlogg/2007/03/30/the-decathlon-of-computer-science/

PolicyHTTPCacheManager/PolicyHTTPCacheManager.py -

  This cache isn't meant to cache anything (sic). The goal is to be
  able to set headers just before the response gets returned.


Conclusions/todo
----------------

* Figure out if the patching of OFS.Image/File is still necessary (to
  associate them with the PolicyHTTPCacheManager). Does zope3 offer
  something here? Does zope 2.8+ already do this?

* Figure out if the IObjectmodified event covers all the cases that
  are now taken care of by the catalog patches. If so, replace those
  patches with an event subscriber that does the same thing. Real easy
  task, probably.

* Does a change in ResourceRegistries also fires such an event? Then
  the resourceregistries patch can also go the way of the dodo.

* Changing the permissions, the roles or the mapping between them:
  does this fire a specific event? Try to do away with the related
  patch, then. First, however, figure out where cachefu uses the
  related counter.

* ``patch_cmf.py`` patches page templates so that 304 get handled and
  response headers get set: isn't that something that can be handled
  with a cache manaager? A cache manager seems to tie in at the
  correct locations. This might make the difference between the
  setting of headers and caching less clear. Hey, I'm even thinking
  about subclassing cachemanagers... I got an email by wiggy about an
  idea he and optilude had: a set of subscribers that can be iterated
  over instead of a one-choice-only event handler.

* Macro caching. Remove it? I'm leaning in that direction. With
  lovely.viewcache, views can be cached. And views can also be
  viewlets: individual pieces of a page. At least for plone 3.0
  that'll be a huge help. I do think that lovely.viewcache could use
  some cachefu UI love: it should be easy to choose certain views and
  identify the cache keys for them. For this, some of the macro code
  might come in real handy.

  Note: We've indeed removed it in trunk (i.e. for 3.0) on the PIKtipi 
  Sprint (tomster, fschulze)

* Figure out if ``CMFSquidTool/queue.py`` really duplicates what
  ``CacheSetup/patch.py`` does: catalog patching. Probably both can
  shed their mortal coil because an IObjectModified event handler is
  the way to go. Yeah, code removal!

* Figure out if python 2.3.5 includes the treading.local library. If
  so, ``CMFSquidTool/threadinglocal.py`` can be cleaned out.

* Investigate zope3's cache managers. What does lovely.viewcache do?
  Are viewcaches a handy way to tie into the publishing process or is
  it a dirty hack to use a cachemanager not to cache something but
  just to evaluate 304 matching and response header setting?

* Can the PolicyHTTPCacheManager be replaced with something
  zope3-like?


