Comment migrer depuis un vieille version de Plone vers la dernière version


La première chose à faire lors de la migration d’un CMS est de regarder si la documentation contient le nécessaire à connaître pour se lancer. La documentation de Plone est très bien fournie avec des conseils pour chaque version majeure depuis la 1.0 :

http://plone.org/documentation/manual/upgrade-guide

 

Il est indiqué quels sont les produits impactés et quels sont les modifications à apporter dans le code tierce.

Il y a deux conseils qui manquent souvent pour le développeur qui manque d’habitude :

  1. Il faut passer par toutes les versions majeures
  2. Dans une même version majeure il faut faire la migration depuis la première version mineure vers la dernière version mineure avant de migrer à la prochaine version majeure id est 3.3 vers 3.3.6 avant de faire 3.3.6 vers 4.0

En suivant ces deux règles supplémentaires il est possible de passer de Plone 2.1 à Plone 4.3 et bientôt

English: The logo of Plone, and Open Source Co...

English: The logo of Plone, and Open Source Content Management System (Photo credit: Wikipedia)

Plone 5 sans avoir trop de surprises.

 

 

 

 

How to get log files rotate in Zope with buildout


— Update : this tip requires Zope 2.11 or Plone 4 —

Zope server includes natively some undocumented option since Zope 2.10 2.11 that allows to rotate log files. They started from the Gilles Lenfant work for iw.rotatezlogs.

The only documenation I found was in ZConfig module test by searching about ‘max-size’ : http://svn.zope.org/ZConfig/trunk/ZConfig/components/logger/tests/test_logger.py?view=markup

To use it in your buildout you should add following code in each intance part :

event-log-custom =
    <logfile>
        path ${buildout:directory}/var/log/instance.log
        #max-size = 5mb
        when D
        old-files 366
    </logfile>

access-log-custom =
    <logfile>
        path ${buildout:directory}/var/log/instance-Z2.log
        #max-size = 5mb
        when D
        old-files 366
    </logfile>

That’s all folks !

Using WingDBG as egg for Zope buildout – updated


For Wing IDE user I start a mercurial branch for the WingDBG product that allow you to make live debugging in the editor in Zope/Plone. It’s just an eggification of the official WingDBG product and stay under the MIT licence.

You can easily add it in your buildout:

[buildout]

eggs += WingDBG
#auto-checkout += WingDBG
find-links += http://plone.org/products/simple

#[sources]
#WingDBG = hg http://bitbucket.org/encolpe/wingdbg

[instance]
zcml += WingDBG
enable-product-installation = on

It was tested successfully under Plone 3.3 and Plone 4 with Wing 3.2.5 and 3.2.6.


Now WingDBG is available as egg for Wing 4.0.1.

2010 : liens de la semaine 1


Cette semaine a été très productive du côté des blogueurs et en particuliers chez les développement autour du core Python.

Python

Pour ceux qui ne connaissent cette boîte de Pandore qu’est le GIL (Global Interpretor Lock) en Python voici un très bon article qui mesure son effet dans le traitement des entrées/sorties avec des systèmes multi-processeurs ou multi-cores :

The Python GIL Visualized

Jesse Noller rappelle les but initiaux du projet Unladen swallow en partie sponsorisé par Google et signale malicieusement en fin d’article qu’une proposition de fusionner leur travail dans Python Core pour Python 3 :

UNLADEN SWALLOW: PYTHON 3’S BEST FEATURE.

Tarek Ziadé, notre président bien aimé, présente les nouvelles fonctionnalités en cours de discussion pour la prochaine version majeure de Distutils, ce qui intéressera tout ceux qui publient des eggs :

Possible new features for Distutils 2.7

Mark Mruss fait une présentation détaillé des ‘docstring’ – ces commentaires qui décorent vos modules, vos classes et vos fonctions – et montre quelle utilisation avancée de celles-ci peut être faite (documentation, tests) :

Introducing Docstrings

Encore un article sur la préparation de la distribution de code qui montre comment mettre en place facilement des scripts  de pré-traitement et de post-traitement :

Using setuptools entry points

La première revue complète du livre « Dive Into Python 3 » (Au cœur de Python 3) :

Book Review: Dive Into Python 3

Enfin deux articles sur les fonctionnalités que les gestionnaires de projets veulent dans mercurial. Dans le premier cas il s’agit de Python-dev qui est censé être migré entièrement cette année sous mercurial :

Where the Hg transition stands

Le deuxième est un développeur qui fait pas mal de Django (vous aimerez sans doute aussi son article free VS free) :

DVCS Ponies

Plone

Voici la description d’un problème récurrent pour les sites multilignues en Plone : comment réordonner des éléments dans une page :

Ordering objects in folders with multilingual sites

Une des solutions les plus faciles à mettre en place est d’afficher toutes les langues lors du tri pour être sur que le Javascript se comporte correctement. L’autre est de demander au Javascript de ne prendre en compte les rangs réels des contenus déplacés plutôt que le rang à l’affichage.

Alex Clarck montre comment installer Plone juste avec buildout en 5 minutes, sans avoir à installer l’unified installer ou Paster :

No, really, you can (just) use Buildout to install Plone

WEB

Encore une fois je terminerai par des liens plus centrés sur le développement Web en général et jQuery en particulier.

Réaliser des animations en jQuery en 7 étapes :

jQuery Animations: A 7-Step Program

Un visualisateur de contenu (d’images) à l’épreuve des balles :

A Bullet-Proof Content Viewer

Personnaliser les événements et l’API des Événements Spéciaux dans jQuery :

Custom Events, and the Special Events API in jQuery

Heads up: new plone.recipe.pound release


Since few weeks plone.recipe.pound 0.5.4 was broken with an output like this:

An internal error occured due to a bug in either zc.buildout or in a recipe being used:
Traceback (most recent call last):
File "/tmp/tmpVcq-b1/zc.buildout-1.2.1-py2.4.egg/zc/buildout/buildout.py", line 1509, in main
File "/tmp/tmpVcq-b1/zc.buildout-1.2.1-py2.4.egg/zc/buildout/buildout.py", line 473, in install
File "/tmp/tmpVcq-b1/zc.buildout-1.2.1-py2.4.egg/zc/buildout/buildout.py", line 1091, in _call
File "/home/encolpe/.virtualenvs/internal/preprod/plone.recipe.pound/plone/recipe/pound/build.py", line 74, in install
installed = CMMIRecipe.install(self)
File "/home/encolpe/.virtualenvs/internal/preprod/downloads/dist/zc.recipe.cmmi-1.2.0/src/zc/recipe/cmmi/__init__.py", line 155, in install
system("make install")
File "/home/encolpe/.virtualenvs/internal/preprod/downloads/dist/zc.recipe.cmmi-1.2.0/src/zc/recipe/cmmi/__init__.py", line 27, in system
raise SystemError("Failed", c)
SystemError: ('Failed', 'make install')

This recipe directly unherit from zc.recipe.cmmi and act as a wrapper around the CMMIRecipe.install method. Most arguments used are transmitted to the recipe are self attributes or keys in self.options. The version 1.2.0 of this recipe publish the May, 18 moved the way that extra arguments are transmitted:

  • zc.recipe.cmmi 1.1.x:  self.options[‘extra_options’]
  • zc.recipe.cmmi 1.2.0: self.extra_options (an empty string by default)

There’s no regression tests, neither an entry in the release history. Youenn did a new release of plone.recipe.pound to fix this new behavior.

Morality:

  • if you are using zc.recipe.cmmi 1.1.x you can still use plone.recipe.pound 0.5.4
  • if you are using zc.recipe.cmmi 1.2.0 or above use plone.recipe.pound 0.5.5
  • other recipes using zc.recipe.cmmi may be broken

Install a maintenance page without restarting apache


These days we are working with fabric to remove some hazard in maintenance.
Recently someone gzip a Data.fs file in production and our customer loose 2 days of work. We choose to use fabric (http://www.nongnu.org/fab/) to limit command line working to the strict minimum.

Now, almost all procedures were added in a fabric factory, from the installation of a development environment to the upgrade of the production server. In this last part, we search a solution to put a maintenance page without have to gain root privileges to restart the apache used in frontend. For static or CGI sites you can use a .htaccess file to override the global rules. For a stopped Zope server it doesn’t help much.

The goal is to use RewriteCond and RewriteRule in a such way that a static HTML file would be displayed when it exists. We can call it ‘maintenance.html‘ and rename it ‘maintenance.html-disabled‘ to let the site be displayed normally.

We start with a virtual host generated by iw.recipe.squid:

<VirtualHost *:80>
   ServerName www.gosseyn.fr

   RewriteEngine On
   RewriteLog /my/path/to/squid/parts/squid/log/rewrite_www.gosseyn.fr.log
   RewriteLogLevel 0

   CustomLog /my/path/to/squid//parts/squid/log/access_www.gosseyn.fr.log common
   ErrorLog /my/path/to/squid//parts/squid/log/error_www.gosseyn.fr.log

   ## common rules for squid rewrite rules
   RewriteRule ^(.*)$ - [E=BACKEND_LOCATION:127.0.0.1]
   RewriteRule ^(.*)$ - [E=BACKEND_PORT:8081]
   RewriteRule ^(.*)$ - [E=BACKEND_PATH:site]

   RewriteRule  ^/(.*)/$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]
   RewriteRule  ^/(.*)$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]

   ## specific rules base on cookie

</VirtualHost>

First we need to declare a document root and to bypass all security rules. We create a new path in the root of our buildout folder called ‘maintenances_pages’.

   DocumentRoot /my/path/to/buildout/prod/maintenance_pages

   <Directory "/my/path/to/buildout/prod/maintenance_pages">
       Options None
       AllowOverride None
       Order allow,deny
       allow from all

       <LimitExcept GET>
           Order allow,deny
           allow from all
       </LimitExcept>
   </Directory>

These declaration should satisfy all paranoid BOFH.
Now create a file called maintenance.html within the folder. All code (javascript, CSS and images) must be embeded, this is a limitation of this approach.
Well, how to detect that a file exist on the filesystem from apache. You can do this with the flag ‘-f’ from the rewrite module:

    RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html -f

Then redirect all pages to our maintenance page:

   RewriteCond %{REQUEST_URI} !/maintenance.html
   RewriteRule $ /maintenance.html [R=302,L]

We also need to disable the standard rewrite rules. The easier way to do it is to use ‘!-f‘.

   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)/$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]
   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]

These rules will not be applied anymore when our maintenance file will be name ‘maintenance.html’. Our goal is reached.

The whole virtualhost file look like this:

<VirtualHost *:80>
   ServerName www.gosseyn.fr

   RewriteEngine On
   RewriteLog /my/path/to/squid/parts/squid/log/rewrite_www.gosseyn.fr.log
   RewriteLogLevel 0

   CustomLog /my/path/to/squid//parts/squid/log/access_www.gosseyn.fr.log common
   ErrorLog /my/path/to/squid//parts/squid/log/error_www.gosseyn.fr.log

   DocumentRoot /my/path/to/buildout/prod/maintenance_pages

   <Directory "/my/path/to/buildout/prod/maintenance_pages">
       Options None
       AllowOverride None
       Order allow,deny
       allow from all

       <LimitExcept GET>
           Order allow,deny
           allow from all
       </LimitExcept>
   </Directory>

   ## our maintenance page
   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html -f
   RewriteCond %{REQUEST_URI} !/maintenance.html
   RewriteRule $ /maintenance.html [R=302,L]

   ## common rules for squid rewrite rules
   RewriteRule ^(.*)$ - [E=BACKEND_LOCATION:127.0.0.1]
   RewriteRule ^(.*)$ - [E=BACKEND_PORT:8081]
   RewriteRule ^(.*)$ - [E=BACKEND_PATH:site]

   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)/$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]
   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]

   ## specific rules base on cookie

</VirtualHost>

It will be difficult to include this in a recipe as we modify an existing virtualhost in a very specific way.

As usual, all feedbacks are appreciated.

How to extend you Plone 3.2 buildout


Plone 3.2 and next comes with a hardcoded versions.cfg for each Plone release. For the current release:
http://dist.plone.org/release/3.2.2/versions.cfg

The Plone Upgrade Guide will show you how to transform a basic buildout created with paster:
http://plone.org/documentation/manual/upgrade-guide/version/upgrading-from-3-x-to-3.2

This solution will work until you already defined a ‘[versions]‘ section in your main buildout file. What we need is to merge the two ‘[versions]‘ sections.

For example, you current ‘buildout.cfg‘ looks like this:

[buildout]
...
eggs =
	archetypes.schematuning
	Products.CacheSetup
	Products.errornumber
	collective.recipe.omelette
	collective.workflowed
	Products.DCWorkflowGraph
	Products.PrintingMailHost
	plone.reload
	Products.PDBDebugMode
	Products.DocFinderTab

versions = versions

[versions]
zope.testing=3.5.1
zope.interface=3.4.1
Products.errornumber=1.2
archetypes.schematuning=1.1
Products.CacheSetup=1.2

If you add extends = http://dist.plone.org/release/3.2.2/versions.cfg it will be overloaded by the local section displayed above and will be simply ignored by ‘bin/buildout‘. If you run an update now all plone bundle eggs will be updated to the last published eggs (Plone 3.3b1, etc).
The solution consist to use the ‘extends’ directive to merge sections. For that we need to put your local ‘[versions]‘ section in a separate file.
dev-versions.cfg‘ will contain:

[versions]
zope.testing=3.5.1
zope.interface=3.4.1
Products.errornumber=1.2
archetypes.schematuning=1.1
Products.CacheSetup=1.2

The ‘buildout.cfg‘ will contain:

[buildout]
...
eggs =
	archetypes.schematuning
	Products.CacheSetup
	Products.errornumber
	collective.recipe.omelette
	collective.workflowed
	Products.DCWorkflowGraph
	Products.PrintingMailHost
	plone.reload
	Products.PDBDebugMode
	Products.DocFinderTab

extends = dev-versions.cfg http://dist.plone.org/release/3.2.2/versions.cfg

versions = versions

We can update now and verify our eggs versions:

  • Plone 3.2.2
  • plone.app.locales 3.2.0
  • Products.CMFPlacefulWorkflow 1.4.0
  • Products.CacheSetup 1.2

It is a certified Plone 3.2.2 installation with our specifics packages.

Fix PIL 1.1.6 installation from sources under debian


Today I found a bug in kupu integration in Plone. I prepared a patch and I tried to reinstall plonenext environment. But this time it fails to install on the PIL egg.

The PIL egg is very dependant to your architecure. It depends of libjpeg, tkinter (tcl/tk), and some other libraries. I’m working with a Debian unstable with 3 versions of tcl/tk for various programs: 8.3, 8.4 and 8.5.

The errors was the following:

getting distribution for ‘PIL’.
libImaging/Effects.c:210: warning: ‘perlin_init’ defined but not used
libImaging/Geometry.c:236: warning: ‘quadratic_transform’ defined but not used
libImaging/Quant.c:311: warning: ‘test_sorted’ defined but not used
libImaging/Quant.c:676: warning: ‘checkContained’ defined but not used
libImaging/QuantHash.c:136: warning: ‘_hashtable_test’ defined but not used
_imagingtk.c:20:16: error: tk.h: Aucun fichier ou répertoire de ce type
_imagingtk.c:23: error: expected ‘)’ before ‘*’ token
_imagingtk.c:31: error: expected specifier-qualifier-list before ‘Tcl_Interp’
_imagingtk.c: In function ‘_tkinit’:
_imagingtk.c:37: error: ‘Tcl_Interp’ undeclared (first use in this function)
_imagingtk.c:37: error: (Each undeclared identifier is reported only once
_imagingtk.c:37: error: for each function it appears in.)
_imagingtk.c:37: error: ‘interp’ undeclared (first use in this function)
_imagingtk.c:45: error: expected expression before ‘)’ token
_imagingtk.c:51: error: ‘TkappObject’ has no member named ‘interp’
_imagingtk.c:55: warning: implicit declaration of function ‘TkImaging_Init’
error: Setup script exited with error: command ‘gcc’ failed with exit status 1

All development libraries was installed, but the installer cannot find /usr/include/tk.h. There are /usr/include/tcl8.3/tk.h, /usr/include/tcl8.4/tk.h, /usr/include/tcl8.5/tk.h. Then I wrote the little patch below for the setup.py file:

Index: setup.py
===================================================================
— setup.py    (révision 460)
+++ setup.py    (copie de travail)
@@ -199,6 +199,9 @@
add_directory(library_dirs, « /usr/lib »)
add_directory(include_dirs, « /usr/include »)

+        if os.path.isfile(os.path.join(‘/usr/include’, ‘tcl’+TCL_VERSION,’tk.h’)):
+            add_directory(include_dirs, os.path.join(‘/usr/include’, ‘tcl’+TCL_VERSION))
+
#
# insert new dirs *before* default libs, to avoid conflicts
# between Python PYD stub libs and real libraries

I hope this patch is generic enough to be merged in the next Imaging release. Waiting this next release, I’m searching for a method to patch an egg before it is compiled by zc.buildout.

Update: this patch is included in PIL 1.1.7beta1

Integration of buildout in your favorite IDE


As Open Source developer I’m concived that we are developping on the best tools ever created (early friday) and I’m using a proprietary IDE… but writen in Python: Wing IDE.
 
With this IDE you need only two operations to make it understand a new buildout:

  • add your bin/instance (bin/zopeinstance or bin/zeoclient) in your project via the project Panel
  • once it is done select it and right click on it to activate it as ‘main debug file’

 
Then wait that wing update the python path for you and enable the autocompletion for your whole project. That’s magical, I love it!

Thanks to Lennart to ask the good question on wing-users mailing list and to Martijn Pieters to give us this excellent recipe.

It would be cool is there’s such recipe fir IDE in the new documentation section on plone.org.