Aviser les clients du besoin de mettre à jour !

Bonjour !

Nous vous écrivons pour vous rappeller que votre serveur virtuel utilise présentement la distribution squeeze, qui aura des mises-à-jour de sécurité jusqu'au 4 mai 2014 [1]. Afin de maintenir l'intégrité de vos systèmes, nous vous recommandons donc de passer à la version suivante, debian Wheezy. Les changements majeurs sont:

• Passage à php 5.4
  • Php 5.3 sera supporté jusqu'en mars 2014 [4], il sera donc possible de garder php 5.3 jusqu'en mars.
• Mysql passe de 5.1 à 5.5 [5]
• Perl passe de 5.10 à 5.14 [6]
• PostgreSQL passe de 8.4 à 9.1
  • Considérer utiliser le backport dans lenny (postgres 9.1) pour éviter la mise à jour de postgres pendant la mise à jour du os

Si vous avez besoin d'aide pour faciliter ces passages, nous pouvons vous aider !

Guillaume !, pour le comité infrastructure

• [1] Debian fourni des mises à jour de sécurité un an après le remplacement de la nouvelle version stable [2] et la dernière version stable est sortie le 4 mai 2013 [3]

  [2] http://www.debian.org/security/faq#lifespan [3] http://www.debian.org/News/2013/20130504 [4] https://wiki.php.net/rfc/php53eol [5] http://dev.mysql.com/doc/refman/5.5/en/upgrading-from-previous-series.html [6] Au minimum, y'a quelqes instructions à: http://blog.timbunce.org/2011/07/21/upgrading-from-perl-5-8/

Upgrade process

Note that this is a shorter form of the release notes process. Also note that this list is explicitly primarily targeted at server platforms and might not apply to desktops. For example, we follow the minimal downtime upgrade procedure.

Voir aussi la procédure d'upgrade des sysadmins de Debian.

Optimisation

1. time the upgrade process
2. first optimisation run: preseeds
3. time again
4. figure out second optimisations
5. time again

Voir 13420 pour ce travail d'optimisation et l'évolution des timings.

Pre-upgrade

1. inform users
2. check to make sure the backup job will not start while the upgrade takes place!
3. check backups on backup server
4. Run in screen and record the session

   • sudo ttyrec -e screen /var/log/upgrade-wheezy.ttyrec

5. Run puppet once to see if there's any outstanding issues. If so, try to fix them.

   • puppet agent -t

6. backup configuration:

   • cd /etc; git tag pre-wheezy
     git gc --prune # make /etc smaller for backup
     tar cfz /var/backups/pre-wheezy-backup.tgz /etc /var/lib/dpkg /var/lib/aptitude/pkgstates
     dpkg --get-selections "*" > /var/backups/dpkg-selections-pre-wheezy.txt

7. Put servers in maintenance in Nagios.

Prepare and check system

Note: keep the following APT repository sources:

 /etc/apt/sources.list.d/aegir-stable.list

 /etc/apt/sources.list.d/koumbit.list

1. disable puppet, pinning and check for packages on hold or broken

   • puppet agent --disable # Disable puppet so it won't overwrite apt's config while we upgrade:
     rm /etc/apt/preferences /etc/apt/preferences.d/* #  Check for pinned (on hold) packages, and possibly disable
     rm /etc/apt/sources.list.d/testing.list # or other similar backports or sources from later releases
     dpkg --get-selections | grep 'hold$'
     aptitude search "~ahold"
     dpkg --audit

   • If you get packages that are held w/ aptitude search "~ahold" unhold them via  aptitude unhold alternc 

2. Run any pending upgrade, remove modutils and make puppet upgradeable

   • apt-get update && apt-get upgrade
     # Unhold the puppet packages in order to upgrade the packages to wheezy correctly
     echo "puppet install" | dpkg --set-selections
     echo "puppet-common install" | dpkg --set-selections
     # Purge modutils - necessary only for machines that date from before lenny
     apt-get purge modutils  

     If modutils is installed, you will get a dependency loop:

     insserv: There is a loop between service munin-node and modutils if stopped
     insserv:  loop involving service modutils at depth 2
     insserv:  loop involving service munin-node at depth 1
     insserv: Stopping modutils depends on munin-node and therefore on system facility `$all' which can not be true!
     insserv: exiting now without changing boot order!
     update-rc.d: error: insserv rejected the script header

Preparing sources.list

1. This mostly means changing squeeze to wheezy

   • sed -i.orig 's/squeeze\(-lts\)\?/wheezy/g;s#/debian-backports#/debian#g' /etc/apt/sources.list
     rm /etc/apt/sources.list.d/squeeze-backports.list
     rm /etc/apt/sources.list.d/squeeze-lts.list

     In vim: :%s/squeeze/wheezy/cg.

   • If you want to track backports for wheezy (although, you probably prefer to let puppet change this for you) you should note that backports are now part of the main debian package repositories. You'll need to change the URL format to this: deb http://debian.mirror.iweb.ca/debian/ wheezy-backports main contrib non-free

     • Note that in some cases, additional sources are defined in /etc/apt/sources.list.d and that you should review these sources to ensure they are still relevant:

       ls /etc/apt/sources.list.d

Upgrading the packages

1. on physical servers: make sure the serial console works

2. Update the package list and check disk space:

   • apt-get update
     apt-get -o APT::Get::Trivial-Only=true dist-upgrade; df -h

     See those tips to claim back disk space if missing

     • On old old old alterncs this will bork, I've just apt-get upgrade alternc and I "didn't get any issues" (yet).
   • Si vous avez ce message d'erreur lors du 'apt-get update' Lecture des listes de paquets... Fait

   W: Aucune clé publique n'est disponible pour la/les clé(s) suivante(s) : 9D6D8F6BC857C906 W: Aucune clé publique n'est disponible pour la/les clé(s) suivante(s) : 7638D0442B90D010

   Prendre soin d'installer les dernières clés de Debian.

   # apt-get install debian-keyring debian-archive-keyring

3. Download packages:

   • apt-get -y -d upgrade && apt-get -y -d dist-upgrade

4. Warn users of potential downtime, if relevant

   • look at the list of packages downloaded above to see if any key service may be disrupted

5. preseed some answers:

   • debconf-set-selections <<EOF
     sysstat sysstat/remove_files    boolean true
     libc6   libraries/restart-without-asking        boolean true
     linux-base linux-base/disk-id-convert-auto boolean true
     linux-base linux-base/disk-id-manual-boot-loader seen true
     postfix postfix/main_mailer_type        select  No configuration
     apt-listchanges apt-listchanges/frontend select mail
     EOF

6. Stop mysql, it can't be stopped after the first minimal upgrade otherwise

   • service mysql stop

7. Minimal upgrade run:

   • env DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=mail apt-get upgrade -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold'

     • we get shown the apt-listchanges, this is not necessary? maybe apt-listchanges/frontend above will fix this. nope it didn't. maybe the env var will fix this? Q&A:

       • Elilo: Configuration de elilo? No sur les vservers, désinstaller

       • Systat: Faut-il supprimer les fichiers de statistiques qui utilisent l'ancien format ? Oui (preseed: sysstat sysstat/remove_files    boolean true

       • libc6: Restart services during package upgrades without asking? Yes (preseed: libc6   libraries/restart-without-asking        boolean true)

       • overwrite /etc/securetty ? No (preseed?? deploy new file with puppet?) (opt2: 16:25)

       • overwrite /etc/backupninja.conf ? no

       • overwrite /etc/apticron/apticron.conf ? no (preseed?? fix checksum?)

       • overwrite /etc/cron-apt/config ? no (preseed: same)

       • update disk UUIDs in config? Yes (preseed: linux-base linux-base/disk-id-convert-auto boolean true, linux-base linux-base/disk-id-manual-boot-loader => seen = true?)

       • overwrite /etc/sudoers ? no (preseed: deployed new version with puppet may need porting if server is in site/site_sudo/files/$fqdn)

       • overwrite /etc/xen/xend-config.sxp ? No

       • overwrite /etc/default/xendomains ? No

       • overwrite /etc/xen-tools/xen-tools.conf ? No

8. On physical servers (dom0): install a new kernel and udev (a 32 bit OR a 64 bit)

   • arch=$(dpkg --print-architecture)
     env DEBIAN_FRONTEND=noninteractive apt-get install  -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' linux-image-$([ "$arch" = "i386" ] && echo "686" || echo "amd64" ) udev

     If the above command fails with an error like this:

     The following packages have unmet dependencies:
      libc6-dev : Breaks: gcc-4.4 (< 4.4.6-4) but 4.4.5-8 is to be installed
     E: Broken packages

     Then you have to force gcc to upgrade before installing the kernel and udev.

     env DEBIAN_FRONTEND=noninteractive apt-get install -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' gcc-4.7

   The main kernel now supports Xen, so we don't need to install a different kernel for dom0s (e.g. you should install the above kernel even with Xen). BUT you need to install a newer version of the hypervisor, else the default option "Linux with Xen Hypervisor" will refuse to boot.

   Make sure to install the right version for both the kernel and the hypervisor, else xen will not work and you might see something like this on the console:

   FATAL: Failed to initialize dom0 state: Invalid argument

   • env DEBIAN_FRONTEND=noninteractive apt-get install -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' xen-hypervisor-4.1-$(dpkg --print-architecture)

9. On dom0 servers: switch the network interfaces config to a manual bridge config. In /etc/network/interfaces, comment out eth0 and replace it with br0, then add a "bridge_ports" line like the following:

   • # The primary network interface
     #allow-hotplug eth0
     #iface eth0 inet static
     auto br0
     iface br0 inet static
             address 199.58.80.xx
             netmask 255.255.255.128
             gateway 199.58.80.xx
             bridge_ports eth0

     Then, edit each .cfg file in /etc/xen/ and change the bridge name from "eth0" (or possibly "xenbr0" in some cases) to "br0" (if no bridge is specified then all should be fine). Finally, edit /etc/xen/xend-config.sxp and comment out the line (network-script network-bridge). The following commands can do the work for you:

   sed -i 's/^\((network-script.*\)$/\#\1/' /etc/xen/xend-config.sxp
   sed -i 's/bridge=eth\(0\|1\)/bridge=br\1/g' /etc/xen/*.cfg

   Si la machine utilise le mode "dual bridge", configurer br1 à la place de eth1 de la même manière que l'on a fait avec eth0

   Si la machine a eth0 de configuré avec puppet, prendre soins de refaire la configuration avec br0 et br1 si nécessaire!

10. On physical servers (dom0):

    • reboot

      Pendant le reboot, utiliser la console série pour aller dans le BIOS et vérifier que la redirection série est configurée à "VT-UTF8" et non "VT-100".

      Une fois la machine repartie, revenir en tant que root et partir une nouvelle session d'enregistrement du tty pour la 2eme phase:

      ttyrec -e screen /var/log/upgrade-wheezy.phase2.ttyrec

11. Warn users of downtime
12. Upgrade the rest of the system:

    • env DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=mail apt-get dist-upgrade -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold'

    • Assemble required md devices required for boot? all
    • Mysql password: keep blank for no change (may happen at least 2x)

    • Postfix: choose "No config" to avoid changes (preseed: postfix postfix/main_mailer_type        select  No configuration)

    • overwrite /etc/monkeysphere/monkeysphere-authentication.conf ? yes (preseed: fixed in puppet )

    • overwrite /etc/nagios/nrpe.cfg no (preseed: move to snippets from puppet)

    • overwrite /etc/logrotate.d/apache2 no? (preseed: not sure)

    • overwrite /etc/apache2/apache.conf yes then move to conf.d

    • overwrite /etc/munin/munin-node.conf no (preseed: fix it up?)

    • overwrite /etc/munin/plugin-conf.d/munin-node yes (preseed: deploy template with puppet instead of removing?)

13. we then want to upgrade facter to 1.7.0, the version from wheezy-backports (since 1.6.10 suffers from a bug with the $is_virtual and $virutal facts). normally wheezy-backports should already be present in /etc/apt/sources.list, but if not you need to add it to the end of that file.

    • apt-get install -t wheezy-backports facter

14. on PHP webservers, purge suhosin and uuid to avoid warnings -- those modules aren't packaged anymore and if they're not purged the cronjob that removes session files complains about being unable to load them:

    • apt-get purge php5-suhosin php5-uuid

Re-enable and run puppet!

1. verify that puppet doesn't break anything:

   • puppet agent --enable; puppet agent -t --noop; puppet agent --disable

2. Re-enable puppet if everything seems ok:

   • puppet agent --enable && puppet agent -t

21:29:27 <@micah> ok, here is a tip for people: if you upgrade to wheezy on your puppetmaster, do update-alternatives --config ruby and set the ruby to 1.8
21:29:50 <@micah> at least if you are using 2.7.18 and our modules
21:30:04 <@micah> because there are things like fqdn_rand that don't work right and its really annoying to fix them

Things to do after the upgrade

1. Install the package koumbit-scripts if it's not already there:

   • apt-get install --no-install-recommends koumbit-scripts

2. check the updates on configuration files, by looking for .dpkg-* or .ucf-* files in /etc, or by using the clean_conflicts script in koumbit-scripts 1.2:

   • /opt/bin/clean_conflicts

3. Reconfigure ssh to ensure that the new ecdsa key is generated

   • dpkg-reconfigure openssh-server

4. On dom0 servers, rerun grub and make sure the right drives are configured for a reboot

   • dpkg-reconfigure grub-pc
     update-grub

     I had some problems during that reboot on ceres.koumbit.net - grub simply wouldn't load, with the following error:

     error: symbol not found: `grub_divmod64_full'.
     Entering rescue mode...

     I had to restore the bootloader using the procedure described in GrubMaintenance. -- TheAnarcat 2013-10-19 11:22:44

     We are experiencing a higher load, from 0.39 on average last year to 2.15 on average during the week after the upgrade, and we suspect this is related to the 3.2 kernel. See 280062. -- TheAnarcat 2013-10-19 11:22:44

5. On domU servers, verify the version of the kernel used. Wheezy kernels are available in dom0, they should be used by the domU.

   • Try switching to pygrub instead of the change below, see XenMaintenance#pygrub.

     When upgrading a domU, make sure it uses the wheezy kernel from its dom0 (hopefully this one was upgraded first). Have a look at /etc/xen/<domU>.cfg

     • -kernel      = '/boot/vmlinuz-2.6.32-5-xen-amd64'
       -ramdisk     = '/boot/initrd.img-2.6.32-5-xen-amd64'
       +kernel      = '/boot/vmlinuz-3.2.0-4-amd64'
       +ramdisk     = '/boot/initrd.img-3.2.0-4-amd64'

     Warning: do NOT do this when upgrading the dom0 if the domUs are not already running wheezy. Some functionality like NFS will not work properly since the new kernel's modules are not present in the domU.

   • You should have the following in the domU.

     uname -a
     Linux ques2 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux

6. On domU and dom0 Reboot one last time and monitor for problems in the boot sequence (lags, or errors, fsck)

   • reboot

7. If applicable, switch from linux-vserver to lxc (Squeeze was the last release supporting LinuxVservers)

8. update the wiki documentation to reflect the new release!

9. Remove obsolete packages

   • apt-get autoremove -y --purge

     • considérer utiliser aptitude purge ?obsolete et pour faire la liste avant de faire le purge aptitude search ?obsolete

10. Cleanup dummy packages:

    • deborphan --guess-dummy

11. Empty apt cache

    • apt-get clean

12. IpChange si ça avait pas été fait pendant l'upgrade à squeeze

13. Vérfier qu'il y a assez d'espace restant pour que les backups aient lieux.

14. Vérifier les stats du serveur sur http://stats.koumbit.net

15. Bien sûr, vérifier que les services offerts par la machine fonctionnent toujours. Tester les vrais services, et aussi vérifier l'état dans nagios.

16. Enlever le downtime dans nagios.

17. Vérifier dans la file roots@rt.k.n si la file est spammé par des erreurs suite à votre mise à jour!

Problèmes rencontrés

Apache

L'upgrade d'apache efface le fichier /etc/apache2/httpd.conf, donc si on conserve le fichier /etc/apache2/apache2.conf tel quel, le service va planter pendant l'upgrade et il faut modifier apache2.conf à la main pour enlever la ligne qui inclus httpd.conf.

Aegir

Lors du dist-upgrade, aegir est marqué comme package qui doit être désinstallé. Il faut upgrader mysql avant de faire le dist-upgrade pour régler la situation.

Selon toute vraissemblance il faut réinstaller /installer Aegir aussi. Voici un log brut des commandes:

apt-get install aegir2
mv /usr/share/drush/commands/provision/remote_import ~/remote_import.bkp
apt-get install aegir2
mv /usr/share/drush/commands/provisionacl ~/provisionacl.bkp
apt-get install aegir2
rm -rf /var/aegir/hostmaster-6.x-2.1/sites/dev0.aegir.koumbit.net/modules/openidadmin
apt-get install aegir2
rm -rf /var/aegir/hostmaster-6.x-2.1/sites/dev0.aegir.koumbit.net/modules/openidadmin/
rm -rf /var/aegir/hostmaster-6.x-1.12/sites/dev0.aegir.koumbit.net/modules/openidadmin/
apt-get install aegir2
apt-get install aegir2
cd /var/aegir/platforms/
ls
cd ..
ls
ls hostmaster-6.x-1.12/sites/
ls hostmaster-6.x-2.1/sites/
rm -rf hostmaster-6.x-2.1/
apt-get install aegir2
mv hostmaster-6.x-1.12/sites/dev0.aegir.koumbit.net/modules/views/ hostmaster-6.x-1.12/sites/all/modules/views
rm -rf hostmaster-6.x-2.1/
apt-get install aegir2
apt-get purge php5-uuid

Voir point suivant.

MySQL

Dans certains cas, il est possible de voir une erreur comme celle ci:

E: Could not perform immediate configuration on 'mysql-client-5.5'. Please see man 5 apt.conf under APT::Immediate-Configure for details. (2)

Ça ne semble pas se produire quand on suit la procédure plus haut à la lettre.

Les release-notes conseillent de rouler apt-get dist-upgrade -o APT::Immediate-Configure=0 pour contourner le problème. Donc essayer ça en premier.

Mais c'est possible que ça ne règle rien. Dans ce cas, deux solutions possibles:

• upgrader mysql manuellement avec apt-get upgrade mysql-client mysql-server avant le dist-upgrade

  • ou bien si les conflits de packages sont trop intenses, on peut utiliser aussi apt-get install mysql-client-5.5 mysql-server-5.5 libc6-dev gcc-4.4

• désinstaller mysql avec apt-get remove mysql-client-5.1, procéder avec le dist-upgrade, puis installer la nouvelle version avec apt-get install mysql-client mysql-server

PHP5.3

Pour garder les paquets Debian pour php 5.3 sur une installation qui sera mise à jour, vous devez ajouter les lignes suivantes dans le fichier /etc/apt/preferences.

la commande pour voir la liste de package php5 installé

# aptitude -t squeeze --disable-columns -F "%p" search ^php5|tr '\n' ' ' 

le contenu de /etc/apt/preferences

Package: libapache2-mod-php5 php-pear php5 php5-adodb php5-auth-pam php5-cgi php5-cli php5-common php5-curl php5-dbg php5-dev php5-ffmpeg php5-gd php5-geoip php5-gmp php5-gpib php5-idn php5-imagick php5-imap php5-interbase php5-json php5-lasso php5-ldap php5-librdf php5-mapscript php5-mcrypt php5-memcache php5-mhash php5-ming php5-mssql php5-mysql php5-odbc php5-pgsql php5-ps php5-pspell php5-radius php5-recode php5-sasl php5-snmp php5-sqlite php5-sqlrelay php5-suhosin php5-sybase php5-symfony php5-symfony1.0 php5-tidy php5-uuid php5-xapian php5-xcache php5-xdebug php5-xmlrpc php5-xsl php-apc
Pin: release n=squeeze
Pin-Priority: 991

Ne pas oublier d'ajouter les packages libapache2-mod-php5, php-pear et php-apc dans la liste.

Note: Drupal 6.x n'est pas compatible avec PHP 5.4.

puppet / lsb_release reports release as n/a

If you see this:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: The release 'n/a' is not yet supported at /etc/puppet/site/site_monkeysphere/manifests/ssh/user.pp:10 on node chartrand.koumbit.net

... it's because lsb_release doesn't recognize the release you're on. It's almost surely because you haven't upgraded the lsb-release package, maybe you tried to run puppet before dist-upgrade was ran? Check the lsb-release version with:

apt-cache policy lsb-release

It should be running the wheezy version (4.1+Debian8).

No more logs

You'll need to run  apt-get install inetutils-syslogd 

References

• Upgrades précédents:

  • SqueezeUpgrade

  • LennyUpgrade

• Upgrade suivant:

  • JessieUpgrade

• http://www.debian.org/releases/wheezy/amd64/release-notes/ch-upgrading.en.html

• known issues and release information

• http://dsa.debian.org/howto/upgrade-to-wheezy/

CategoryDebian