Product Opener/Installation/Debian or Ubuntu

Also see Product Opener Installation

This page lists all the steps necessary to install Product Opener on Debian, including a development environment.

Installation on Jessie

We start from a clean Jessie install.

Get the code and configure it

apt-get install git

cd /srv
git clone .../openfoodfacts/
mkdir /srv/openfoodfacts/users
mkdir /srv/openfoodfacts/products
chown -R www-data:www-data /srv/openfoodfacts

cd /srv/openfoodfacts/cgi
rm Blogs
ln -s . Blogs
ln -s SiteLang_off.pm SiteLang.pm

Edit a few variables in Config2.pm:

$server_domain = "yourdomain.com";
$www_root="/srv/openfoodfacts/html";
$data_root="/srv/openfoodfacts";

Also in Config_off.pm:

$domain = 'yourdomain.com';
$contact_email = 'webmaster@yourdomain.com';
$admin_email = 'webmaster@yourdomain.com';

And in startup.pl:

use lib "/srv/openfoodfacts/cgi/";

In /srv/openfoodfacts/cgi/Display.pm, swap lines 3750-3752 and 3769-3771 (location of Javascript things)

Install packaged prerequisites

apt-get install mongodb memcached imagemagick tesseract-ocr tesseract-ocr-fra geoip-bin geoip-database libwww-perl libimage-magick-perl libxml-encoding-perl libtext-unaccent-perl libmime-lite-perl libcache-memcached-fast-perl libjson-perl libclone-perl  libgraphviz-perl libmime-lite-perl libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl libxml-feedpp-perl libmongodb-perl liburi-find-perl liburi-escape-xs-perl libtest-nowarnings-perl libhtml-defang-perl libgeo-ip-perl libdatetime-format-mail-perl libperl-dev zlib1g-dev apache2 build-essential

Prepare CPAN

If you never used CPAN before, you can automatically configure it:

(echo y;echo o conf prerequisites_policy follow;echo o conf commit)|cpan

Install a few Perl modules from CPAN

cpan Net::IDN::Punycode
cpan Encode::Punycode
cpan GraphViz2
cpan Image::OCR::Tesseract

Rebuild Apache 2.2 and mod_perl

cd /usr/local/src/
wget http://apache.crihan.fr/dist//httpd/httpd-2.2.29.tar.gz
tar xvfz httpd-2.2.29.tar.gz
cd httpd-2.2.29
./configure --with-mpm=prefork --prefix=/usr/local  --enable-rewrite --enable-proxy --enable-proxy_http --enable-deflate --disable-userdir --enable-headers
make
make install

cd /usr/local/src/
wget http://mir2.ovh.net/ftp.apache.org/dist/perl/mod_perl-2.0.8.tar.gz
tar xvfz mod_perl-2.0.8.tar.gz
cd mod_perl-2.0.8
perl Makefile.PL MP_APXS=/usr/local/bin/apxs
make
make install

cat >> /usr/local/conf/httpd.conf <<EOF
Listen 8001

LoadModule perl_module modules/mod_perl.so

PerlWarn Off
PerlRequire /srv/openfoodfacts/cgi/startup.pl

<VirtualHost *:8001>
  DocumentRoot /srv/openfoodfacts/html
  ServerName yourdomain.com
  ErrorLog /var/log/apache2/error.log
  CustomLog /var/log/apache2/access.log combined
  ScriptAlias /cgi/ /srv/openfoodfacts/cgi/

  <Location /cgi>
    SetHandler perl-script
    PerlResponseHandler ModPerl::Registry
    PerlOptions +ParseHeaders
    Options +ExecCGI
    Order allow,deny
    Allow from all
  </Location>
</VirtualHost>

PerlPostReadRequestHandler My::ProxyRemoteAddr
EOF

sed -i -e 's/^Listen 80$/#Listen 80/' /usr/local/conf/httpd.conf
sed -i -e 's/^User.*/User www-data/' /usr/local/conf/httpd.conf
sed -i -e 's/^Group.*/Group www-data/' /usr/local/conf/httpd.conf

/usr/local/bin/apachectl -k start

Configure standard Apache 2

a2enmod proxy_http
a2enmod rewrite

cat >> /etc/apache2/sites-available/product-opener-proxy.conf <<EOF
<VirtualHost *>
  DocumentRoot /srv/openfoodfacts/html
  ServerName yourdomain.com
  ServerAlias *.yourdomain.com

  <Directory "/srv/openfoodfacts/html">
    Options Indexes FollowSymLinks
    Require all granted
  </Directory>
  ProxyPreserveHost On
  RewriteEngine on
  RewriteCond  %{REQUEST_URI}  !/./
  RewriteRule ^(/cgi/.*)$ http://localhost:8001$1 [P,L]
  RewriteMap escape int:escape
  RewriteRule ^/favicon.ico$ /favicon.ico [L]
  RewriteCond  %{REQUEST_URI}  !^/images/
  RewriteCond  %{REQUEST_URI}  !^/js/
  RewriteCond  %{REQUEST_URI}  !^/rss/
  RewriteCond  %{REQUEST_URI}  !^/robots
  RewriteCond  %{REQUEST_URI}  !^/clicks/
  RewriteCond  %{REQUEST_URI}  !^/data/
  RewriteCond  %{REQUEST_URI}  !^/files/
  RewriteRule  ^(.*)$ http://localhost:8001/cgi/display.pl?${escape:$1} [P,L,QSA]
</VirtualHost>
EOF

a2ensite product-opener-proxy.conf
a2dissite 000-default.conf
service apache2 restart

You should now have a working Productopener instance running at http://yourdomain.com/. Enjoy!

Old notes

All instructions below are for user "stephane", please use your own name. :-)

Open Food Facts test / development server

OFF test / development server is a small dedicated server ("kimsufi 2G", the cheap sub-brand of OVH).

• Linux ks3095298.kimsufi.com 3.10.23-xxxx-std-ipv6-64 #1 SMP Tue Mar 18 14:48:24 CET 2014 x86_64 GNU/Linux
• CPU: Intel(R) Atom(TM) CPU 230 @ 1.60GHz - Cores: 2 - Cache: 512KB
• RAM: 2 Gb
• Disk: 1 Tb

Debian install

Fresh Debian install with OVH / kimsufi install process:

• Step 1:
  • OS - Type: Basic
  • Debian 7.5 stable (Wheezy) - debian 7 - 64 bits
• Step 3: Partitions
  • Default:
    • / ext4 20 Gb
    • /home/ ext4 979.5 Gb
    • swap 1 x 512 Mb

DNS

ks3095298.kimsufi.com IP address: 94.23.195.82

Create 2 A records in the DNS zone of the domain you want to use:

• openfoodfacts.ovh 94.23.195.82
• *.openfoodfacts.ovh 94.23.195.82

Install

Update Debian

Usually done by OVH Debian install:

apt-get update
apt-get upgrade

Security

Install fail2ban and sudo:

apt-get install fail2ban
apt-get install sudo

Create an user and give it sudo access:

adduser stephane
adduser stephane sudo

Install software used by Product Opener

apt-get install mongodb

apt-get install memcached

apt-get install imagemagick

apt-get install tesseract-ocr
apt-get install tesseract-ocr-fra

apt-get install exim4
dpkg-reconfigure exim4-config
- Internet Site mail is sent by smtp  
- 127.0.0.1

apt-get install geoip-bin geoip-database

Install libraries used by Product Opener

apt-get install zlib1g-dev

Install development tools

Note: the Product Opener code is currently on a private bitbucket.org repository and is in the process of being made publicly available (open source). If you want to help us to clean / document Product Opener to open it, request access to the bitbucket repository: stephane@openfoodfacts.org

apt-get install build-essential
apt-get install git

git clone https://stephane@bitbucket.org/openfoodfacts/product-opener.git

(get the link for your user on bitbucket.org)

Apache 2.2 + mod_perl: Product Opener backend

Build

apt-get install libperl-dev

cd /home/stephane
mkdir src
cd src
wget http://apache.crihan.fr/dist//httpd/httpd-2.2.29.tar.gz
tar xvfz httpd-2.2.29.tar.gz
cd httpd-2.2.29
./configure --with-mpm=prefork --prefix=/home/stephane/apache  --enable-rewrite --enable-proxy --enable-proxy_http --enable-deflate  --disable-userdir --enable-headers
make
make install
cd ~

cd ~/src
wget http://mir2.ovh.net/ftp.apache.org/dist/perl/mod_perl-2.0.8.tar.gz 
tar xvfz mod_perl-2.0.8.tar.gz
cd mod_perl-2.0.8
perl Makefile.PL MP_APXS=/home/stephane/apache/bin/apxs 
make
make install
cd ~

Configuration

In ~/apache/conf/httpd.conf:

Change the port to something else than 80 (the reverse proxy will listen on 80 and forward to the backend on another port).

Listen 8001

If you are installing several Product Openers on the same machine, use different ports.

Change the Apache user to your user:

User stephane
Group stephane

Add to apache/conf/httpd.conf:

LoadModule perl_module modules/mod_perl.so

PerlWarn Off
PerlRequire /home/stephane/product-opener/cgi/startup.pl


<Location /cgi>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
PerlOptions +ParseHeaders
Options +ExecCGI
Order allow,deny
Allow from all
</Location>


<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName openfoodfacts.ovh
ErrorLog /home/stephane/logs/error_log
CustomLog /home/stephane/logs/access_log combined
ScriptAlias /cgi/ "/home/stephane/product-opener/cgi/"
</VirtualHost>

PerlPostReadRequestHandler My::ProxyRemoteAddr

Apache 2.4 - light weight reverse proxy

Build

apt-get install libpcre3 libpcre3-dev

cd /home/stephane/src
wget http://apache.websitebeheerjd.nl//httpd/httpd-2.4.12.tar.gz
tar xvfz httpd-2.4.12.tar.gz
cd httpd-2.4.12

cd srclib
wget http://mir2.ovh.net/ftp.apache.org/dist//apr/apr-1.5.1.tar.gz
wget http://mir2.ovh.net/ftp.apache.org/dist//apr/apr-util-1.5.4.tar.gz   
tar xvfz apr-1.5.1.tar.gz
tar xvfz apr-util-1.5.4.tar.gz 
ln -s apr-1.5.1 apr
ln -s apr-util-1.5.4 apr-util
cd .. 

./configure --prefix=/home/stephane/proxy --enable-rewrite --enable-proxy --enable-proxy_http --disable-userdir --enable-headers --enable-expires --enable-mime --enable-so --with-included-apr

make
sudo make install
cd ~

Configuration

in ~/proxy/conf/httpd.conf :

• uncomment #LoadModule rewrite_module modules/mod_rewrite.so
• uncomment #LoadModule slotmem_shm_module modules/mod_slotmem_shm.so

Add in httpd.conf :

<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName world.openfoodfacts.ovh
ServerAlias *.openfoodfacts.ovh

ErrorLog /home/stephane/logs/proxy_error_log
CustomLog /home/stephane/logs/proxy_access_log combined
ServerAdmin stephane@openfoodfacts.org

<Location "/foundation/foundation-icons" >
Header set Access-Control-Allow-Origin "*"
</Location>

<Location "/images/products" >
Header set Link "<http://creativecommons.org/licenses/by-sa/3.0/>; rel='license'; title='CC-BY-SA 3.0'"
</Location>


<Directory "/home/stephane/product-opener/html">
    Options Indexes FollowSymLinks
    Require all granted
</Directory>
ProxyPreserveHost On
RewriteEngine on
RewriteCond  %{REQUEST_URI}  !/./
RewriteRule ^(/cgi/.*)$ http://localhost:8001$1 [P,L]
RewriteMap escape int:escape
RewriteRule ^/favicon.ico$ /favicon.ico [L]
RewriteCond  %{REQUEST_URI}  !^/images/
RewriteCond  %{REQUEST_URI}  !^/js/
RewriteCond  %{REQUEST_URI}  !^/rss/
RewriteCond  %{REQUEST_URI}  !^/robots
RewriteCond  %{REQUEST_URI}  !^/clicks/
RewriteCond  %{REQUEST_URI}  !^/data/
RewriteCond  %{REQUEST_URI}  !^/files/
RewriteCond  %{REQUEST_URI}  !^/resources/
RewriteCond  %{REQUEST_URI}  !^/foundation/
RewriteRule  ^(.*)$ http://localhost:8001/cgi/display.pl?${escape:$1} [P,L,QSA]
</VirtualHost>


<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName openfoodfacts.ovh
ErrorLog /home/stephane/logs/proxy_error_log
CustomLog /home/stephane/logs/proxy_access_log combined
DirectoryIndex index.html index.shtml
<Directory "/home/stephane/product-opener/html">
    Options -Indexes +FollowSymLinks +Includes
    Require all granted
</Directory>
RewriteEngine on
RewriteCond %{HTTP_HOST} ^stephane\.openfoodfacts\.ovh
RewriteRule ^/products$ /products.shtml [L]

</VirtualHost>

Install Perl modules

If possible, use apt-get to install packages directly. If they are not available, you can use cpan to install the modules.

apt-get install libwww-perl libimage-magick-perl 
apt-get install  libxml-encoding-perl libtext-unaccent-perl libmime-lite-perl
apt-get install libcache-memcached-fast-perl libjson-perl libclone-perl  libgraphviz-perl 
apt-get install libmime-lite-perl  # needs to be done after installing exim4
apt-get install libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl 
apt-get install libxml-feedpp-perl libmongodb-perl liburi-find-perl

Modules that could not be installed using apt-get:

cpan
install URI::Escape::XS
install Image::OCR::Tesseract # needs imagemagick installed first

Modules that HAVE to be installed using cpan:

As of Feb 3rd 2015, the MongoDB perl module installed through "apt-get install libmongodb-perl" is too old (0.45):

perl -MMongoDB -e 'print $MongoDB::VERSION . "\n"'
0.45

This causes an error when trying to access a list of "tags" such as categories:

Software error:
Not an ARRAY reference at /home/stephane/product-opener/cgi/Blogs/Display.pm line 797.

A newer MongoDB perl module needs to be installed with cpan.

cpan
install MongoDB

perl -MMongoDB -e 'print $MongoDB::VERSION . "\n"'
v0.708.0.0

cgi/startup.pl preloads a lot of modules, you can check that those modules are installed by running:

cd /home/stephane/product-opener/cgi
perl startup.pl

Create some paths and links

cd /home/stephane/product-opener/cgi
ln -s ./ Blogs
ln -s S