Product Opener/Installation/Debian or Ubuntu

From Open Food Facts wiki
Revision as of 09:29, 27 May 2016 by Stephane (talk | contribs) (install on jessie)
Jump to navigation Jump to search

Also see Product Opener Installation This page lists all the steps necessary to install Product Opener on Debian, including a development environment.

Installation on Jessie

We start from a clean Jessie install.

Get the code and configure it

apt-get install git

git clone <repo> /srv/openfoodfacts
mkdir /srv/openfoodfacts/users
mkdir /srv/openfoodfacts/products
chown -R www-data:www-data /srv/openfoodfacts

cd /srv/openfoodfacts/cgi
rm Blogs
ln -s . Blogs
ln -s SiteLang_off.pm SiteLang.pm

Edit a few variables in Config2.pm: 

$server_domain = "yourdomain.com";
$www_root="/srv/openfoodfacts/html";
$data_root="/srv/openfoodfacts";

Also in Config_off.pm: 

$domain = 'yourdomain.com';
$contact_email = 'webmaster@yourdomain.com';
$admin_email = 'webmaster@yourdomain.com';

And in startup.pl: 

use lib "/srv/openfoodfacts/cgi/";

Install packaged prerequisites

apt-get install apache2 build-essential geoip-bin geoip-database imagemagick libbarcode-zbar-perl libcache-memcached-fast-perl libclone-perl libcrypt-passwdmd5-perl libdatetime-format-mail-perl \
libencode-detect-perl libgeo-ip-perl libgraphics-color-perl libgraphviz-perl libhtml-defang-perl libimage-magick-perl libjson-perl libmime-lite-perl libmime-lite-perl libmongodb-perl libperl-dev \
libtest-nowarnings-perl libtext-unaccent-perl liburi-escape-xs-perl liburi-find-perl libwww-perl libxml-encoding-perl libxml-feedpp-perl memcached mongodb tesseract-ocr tesseract-ocr-fra zlib1g-dev

Prepare CPAN

If you never used CPAN before, you can automatically configure it: 

(echo y;echo o conf prerequisites_policy follow;echo o conf commit)|cpan

Install a few Perl modules from CPAN

cpan Net::IDN::Punycode
cpan Encode::Punycode
cpan GraphViz2
cpan Image::OCR::Tesseract
cpan Algorithm::CheckDigits
cpan XML::Simple

Note: on Debian Jessie, you might need to install the follwoing to be able to get cpan to install GraphViz2 

apt-get install libtest-version-perl

Rebuild Apache 2.2 and mod_perl

cd /usr/local/src/
wget http://apache.crihan.fr/dist/httpd/httpd-2.2.31.tar.gz
tar xvfz httpd-2.2.31.tar.gz
cd httpd-2.2.31
./configure --with-mpm=prefork --prefix=/usr/local  --enable-rewrite --enable-proxy --enable-proxy_http --enable-deflate --disable-userdir --enable-headers
make
make install

cd /usr/local/src/
wget http://mir2.ovh.net/ftp.apache.org/dist/perl/mod_perl-2.0.9.tar.gz
tar xvfz mod_perl-2.0.9.tar.gz
cd mod_perl-2.0.9
perl Makefile.PL MP_APXS=/usr/local/bin/apxs
make
make install

cat >> /usr/local/conf/httpd.conf <<EOF
Listen 8001

LoadModule perl_module modules/mod_perl.so

PerlWarn Off
PerlRequire /srv/openfoodfacts/cgi/startup.pl

<VirtualHost *:8001>
  DocumentRoot /srv/openfoodfacts/html
  ServerName yourdomain.com
  ErrorLog /var/log/apache2/error.log
  CustomLog /var/log/apache2/access.log combined
  ScriptAlias /cgi/ /srv/openfoodfacts/cgi/

  <Location /cgi>
    SetHandler perl-script
    PerlResponseHandler ModPerl::Registry
    PerlOptions +ParseHeaders
    Options +ExecCGI
    Order allow,deny
    Allow from all
  </Location>
</VirtualHost>

PerlPostReadRequestHandler My::ProxyRemoteAddr
EOF

sed -i -e 's/^Listen 80$/#Listen 80/' /usr/local/conf/httpd.conf
sed -i -e 's/^User.*/User www-data/' /usr/local/conf/httpd.conf
sed -i -e 's/^Group.*/Group www-data/' /usr/local/conf/httpd.conf

/usr/local/bin/apachectl -k start

Configure standard Apache 2

a2enmod proxy_http
a2enmod rewrite

cat >> /etc/apache2/sites-available/product-opener-proxy.conf <<EOF
<VirtualHost *>
  DocumentRoot /srv/openfoodfacts/html
  ServerName yourdomain.com
  ServerAlias *.yourdomain.com

  <Directory "/srv/openfoodfacts/html">
    Options Indexes FollowSymLinks
    Require all granted
  </Directory>
  ProxyPreserveHost On
  RewriteEngine on
  RewriteCond  %{REQUEST_URI}  !/./
  RewriteRule ^(/cgi/.*)$ http://localhost:8001$1 [P,L]
  RewriteMap escape int:escape
  RewriteRule ^/favicon.ico$ /favicon.ico [L]
  RewriteCond  %{REQUEST_URI}  !^/images/
  RewriteCond  %{REQUEST_URI}  !^/js/
  RewriteCond  %{REQUEST_URI}  !^/foundation/
  RewriteCond  %{REQUEST_URI}  !^/rss/
  RewriteCond  %{REQUEST_URI}  !^/robots
  RewriteCond  %{REQUEST_URI}  !^/clicks/
  RewriteCond  %{REQUEST_URI}  !^/data/
  RewriteCond  %{REQUEST_URI}  !^/files/
  RewriteRule  ^(.*)$ http://localhost:8001/cgi/display.pl?${escape:$1} [P,L,QSA]
</VirtualHost>
EOF

a2ensite product-opener-proxy.conf
a2dissite 000-default.conf
service apache2 restart

You should now have a working Productopener instance running at http://yourdomain.com/. Enjoy!

Installation on Debian Jessie with stock Apache and nginx

Vous êtes sur le point d'installer un OS sur le serveur ks3095298.kimsufi.com. Choisissez le type d'installation que vous souhaitez utiliser :

-> Installer à partir d'un template OVH

Debian 8.1 stable (Jessie) (stable) - debian8

root@ks3095298:~# uname -a
Linux ks3095298.kimsufi.com 3.14.32-xxxx-grs-ipv6-64 #5 SMP Wed Sep 9 17:24:34 CEST 2015 x86_64 GNU/Linux

 perl -v

This is perl 5, version 20, subversion 2 (v5.20.2) built for x86_64-linux-gnu-thread-multi


apt-get update
apt-get upgrade

apt-get install fail2ban
apt-get install sudo

apt-get install mongodb

MongoDB shell version: 2.4.10

vi /etc/mongodb.conf
- change dbpath
- uncomment :
#port = 27017
#nohttpinterface = true


apt-get install git

apt-get install apache2
apt-get install libapache2-mod-perl2
apt-get install modperl

adduser off

apt-get install imagemagick
apt-get install memcached
apt-get install tesseract-ocr tesseract-ocr-fra
apt-get install exim4
dpkg-reconfigure exim4-config
apt-get install geoip-bin geoip-database
apt-get install zlib1g-dev

apt-get install libwww-perl libimage-magick-perl libxml-encoding-perl libtext-unaccent-perl libmime-lite-perl libcache-memcached-fast-perl libjson-perl libclone-perl  libgraphviz-perl libmime-lite-perl libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl libxml-feedpp-perl libmongodb-perl liburi-find-perl libxml-simple-perl


cp -a /home/openfoodfacts-bitbucket/product-opener/cgi /home/off/cgi/

/home/off/cgi# rm Blogs
root@ks3095298:/home/off/cgi# ln -s /home/off/cgi Blogs

ln -s SiteLang_off.pm SiteLang.pm

vi /home/off/startup.pl
# Needs to be configured
use lib "/home/off/cgi/";


ln -s ../sites-available/001-off.conf  001-off.conf
service apache2 restart

To see errors, missing perl modules etc.

/var/log/apache2# tail -f error.log

cpan
install URI::Escape::XS
install Encode::Punycode
install GraphViz2
install HTML::Defang
install Algorithm::CheckDigits
install Geo::IP
install Image::OCR::Tesseract
DateTime::Format::Mail
DateTime::Format::CLDR
DateTime::Locale




Issues:
    
MongoDB does not start:
    
    service mongodb start
root@ks3095298:/home/off/cgi# ps -ef | grep mongo
root     21814 23471  0 23:56 pts/0    00:00:00 grep mongo


in /var/log/syslog:
    
    Jan 11 00:05:05 ks3095298 systemd[1]: Starting An object/document-oriented database...
Jan 11 00:05:05 ks3095298 systemd[1]: Started An object/document-oriented database.
Jan 11 00:05:05 ks3095298 mongod[22404]: Mon Jan 11 00:05:05.434 terminate() called, printing stack (if implemented for platform):
Jan 11 00:05:05 ks3095298 mongod[22404]: 0xa6d3c6 0x57e204 0x7212186b7bb6 0x7212186b7c01 0x7212186b7e19 0x7212192b666d 0xa6070d 0xa61a67 0xa5f8ca 0xa5f979 0x5a626c 0x5a65d2 0x5a6a93 0x5a6ab9 0x5a6e5c 0x56647b 0x721217dbab45 0x57e05c
Jan 11 00:05:05 ks3095298 mongod[22404]: /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xa6d3c6]


sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com --recv EA312927


Text::Unaccent was causing Apache to segfault, replaced the module with pure perl

remote_ip has been replaced in Apache 2.4, need to check if we do get the X-Forwarder-For ip directly now


Loading products from /products in the mongodb database:
    
update_all_products_from_dir_in_mongodb.pl
    
    

Trying to use nginx for the reverse proxy:
    
apt-get install nginx   (need port 80 to be available, stop apache or change its port first)



root@ks3095298:/etc/nginx/sites-enabled# more /etc/nginx/sites-available/off
##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# http://wiki.nginx.org/Pitfalls
# http://wiki.nginx.org/QuickStart
# http://wiki.nginx.org/Configuration
#
# Generally, you will want to move this file somewhere, and start with a clean
# file but keep this around for reference. Or just disable in sites-enabled.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##

# Default server configuration
#
server {
        listen 80 default_server;
        listen [::]:80 default_server;

        server_name openfoodfacts.ovh *.openfoodfacts.ovh;

        # SSL configuration
        #
        # listen 443 ssl default_server;
        # listen [::]:443 ssl default_server;
        #
        # Self signed certs generated by the ssl-cert package
        # Don't use them in a production server!
        #
        # include snippets/snakeoil.conf;

        root /home/off/html;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        location ~ ^/(images|js|rss|data|files|resources|foundation)/ {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }

        location = /robots.txt {
                try_files $uri $uri/ =404;
        }

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://127.0.0.1:8001/cgi/display.pl?;
        }

        location /cgi/ {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://127.0.0.1:8001;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #       deny all;
        #}
}

Old notes

All instructions below are for user "stephane", please use your own name. :-)

Open Food Facts test / development server

OFF test / development server is a small dedicated server ("kimsufi 2G", the cheap sub-brand of OVH).

  • Linux ks3095298.kimsufi.com 3.10.23-xxxx-std-ipv6-64 #1 SMP Tue Mar 18 14:48:24 CET 2014 x86_64 GNU/Linux
  • CPU: Intel(R) Atom(TM) CPU 230 @ 1.60GHz - Cores: 2 - Cache: 512KB
  • RAM: 2 Gb
  • Disk: 1 Tb

Debian install

Fresh Debian install with OVH / kimsufi install process:

  • Step 1:
    • OS - Type: Basic
    • Debian 7.5 stable (Wheezy) - debian 7 - 64 bits
  • Step 3: Partitions
    • Default:
      • / ext4 20 Gb
      • /home/ ext4 979.5 Gb
      • swap 1 x 512 Mb

DNS

ks3095298.kimsufi.com IP address: 94.23.195.82

Create 2 A records in the DNS zone of the domain you want to use:

  • openfoodfacts.ovh 94.23.195.82
  • *.openfoodfacts.ovh 94.23.195.82

Install

Update Debian

Usually done by OVH Debian install: 

apt-get update
apt-get upgrade

Security

Install fail2ban and sudo: 

apt-get install fail2ban
apt-get install sudo

Create an user and give it sudo access: 

adduser stephane
adduser stephane sudo


Install software used by Product Opener

apt-get install mongodb

apt-get install memcached

apt-get install imagemagick

apt-get install tesseract-ocr
apt-get install tesseract-ocr-fra

apt-get install exim4
dpkg-reconfigure exim4-config
- Internet Site mail is sent by smtp  
- 127.0.0.1

apt-get install geoip-bin geoip-database

Install libraries used by Product Opener

apt-get install zlib1g-dev


Install development tools

Note: the Product Opener code is currently on a private bitbucket.org repository and is in the process of being made publicly available (open source). If you want to help us to clean / document Product Opener to open it, request access to the bitbucket repository: stephane@openfoodfacts.org 

apt-get install build-essential
apt-get install git

git clone https://stephane@bitbucket.org/openfoodfacts/product-opener.git

(get the link for your user on bitbucket.org)


Apache 2.2 + mod_perl: Product Opener backend

Build

apt-get install libperl-dev

cd /home/stephane
mkdir src
cd src
wget http://apache.crihan.fr/dist//httpd/httpd-2.2.29.tar.gz
tar xvfz httpd-2.2.29.tar.gz
cd httpd-2.2.29
./configure --with-mpm=prefork --prefix=/home/stephane/apache  --enable-rewrite --enable-proxy --enable-proxy_http --enable-deflate  --disable-userdir --enable-headers
make
make install
cd ~

cd ~/src
wget http://mir2.ovh.net/ftp.apache.org/dist/perl/mod_perl-2.0.8.tar.gz 
tar xvfz mod_perl-2.0.8.tar.gz
cd mod_perl-2.0.8
perl Makefile.PL MP_APXS=/home/stephane/apache/bin/apxs 
make
make install
cd ~

Configuration

In ~/apache/conf/httpd.conf:

Change the port to something else than 80 (the reverse proxy will listen on 80 and forward to the backend on another port). 

Listen 8001

If you are installing several Product Openers on the same machine, use different ports.

Change the Apache user to your user: 

User stephane
Group stephane

Add to apache/conf/httpd.conf: 

LoadModule perl_module modules/mod_perl.so

PerlWarn Off
PerlRequire /home/stephane/product-opener/cgi/startup.pl


<Location /cgi>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
PerlOptions +ParseHeaders
Options +ExecCGI
Order allow,deny
Allow from all
</Location>


<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName openfoodfacts.ovh
ErrorLog /home/stephane/logs/error_log
CustomLog /home/stephane/logs/access_log combined
ScriptAlias /cgi/ "/home/stephane/product-opener/cgi/"
</VirtualHost>

PerlPostReadRequestHandler My::ProxyRemoteAddr

Apache 2.4 - light weight reverse proxy

Build

apt-get install libpcre3 libpcre3-dev

cd /home/stephane/src
wget http://apache.websitebeheerjd.nl//httpd/httpd-2.4.12.tar.gz
tar xvfz httpd-2.4.12.tar.gz
cd httpd-2.4.12

cd srclib
wget http://mir2.ovh.net/ftp.apache.org/dist//apr/apr-1.5.1.tar.gz
wget http://mir2.ovh.net/ftp.apache.org/dist//apr/apr-util-1.5.4.tar.gz   
tar xvfz apr-1.5.1.tar.gz
tar xvfz apr-util-1.5.4.tar.gz 
ln -s apr-1.5.1 apr
ln -s apr-util-1.5.4 apr-util
cd .. 

./configure --prefix=/home/stephane/proxy --enable-rewrite --enable-proxy --enable-proxy_http --disable-userdir --enable-headers --enable-expires --enable-mime --enable-so --with-included-apr

make
sudo make install
cd ~

Configuration

in ~/proxy/conf/httpd.conf :

  • uncomment #LoadModule rewrite_module modules/mod_rewrite.so
  • uncomment #LoadModule slotmem_shm_module modules/mod_slotmem_shm.so

Add in httpd.conf : 

<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName world.openfoodfacts.ovh
ServerAlias *.openfoodfacts.ovh

ErrorLog /home/stephane/logs/proxy_error_log
CustomLog /home/stephane/logs/proxy_access_log combined
ServerAdmin stephane@openfoodfacts.org

<Location "/foundation/foundation-icons" >
Header set Access-Control-Allow-Origin "*"
</Location>

<Location "/images/products" >
Header set Link "<http://creativecommons.org/licenses/by-sa/3.0/>; rel='license'; title='CC-BY-SA 3.0'"
</Location>


<Directory "/home/stephane/product-opener/html">
    Options Indexes FollowSymLinks
    Require all granted
</Directory>
ProxyPreserveHost On
RewriteEngine on
RewriteCond  %{REQUEST_URI}  !/./
RewriteRule ^(/cgi/.*)$ http://localhost:8001$1 [P,L]
RewriteMap escape int:escape
RewriteRule ^/favicon.ico$ /favicon.ico [L]
RewriteCond  %{REQUEST_URI}  !^/images/
RewriteCond  %{REQUEST_URI}  !^/js/
RewriteCond  %{REQUEST_URI}  !^/rss/
RewriteCond  %{REQUEST_URI}  !^/robots
RewriteCond  %{REQUEST_URI}  !^/clicks/
RewriteCond  %{REQUEST_URI}  !^/data/
RewriteCond  %{REQUEST_URI}  !^/files/
RewriteCond  %{REQUEST_URI}  !^/resources/
RewriteCond  %{REQUEST_URI}  !^/foundation/
RewriteRule  ^(.*)$ http://localhost:8001/cgi/display.pl?${escape:$1} [P,L,QSA]
</VirtualHost>


<VirtualHost *>
DocumentRoot /home/stephane/product-opener/html
ServerName openfoodfacts.ovh
ErrorLog /home/stephane/logs/proxy_error_log
CustomLog /home/stephane/logs/proxy_access_log combined
DirectoryIndex index.html index.shtml
<Directory "/home/stephane/product-opener/html">
    Options -Indexes +FollowSymLinks +Includes
    Require all granted
</Directory>
RewriteEngine on
RewriteCond %{HTTP_HOST} ^stephane\.openfoodfacts\.ovh
RewriteRule ^/products$ /products.shtml [L]

</VirtualHost>

Install Perl modules

If possible, use apt-get to install packages directly. If they are not available, you can use cpan to install the modules. 

apt-get install libwww-perl libimage-magick-perl 
apt-get install  libxml-encoding-perl libtext-unaccent-perl libmime-lite-perl
apt-get install libcache-memcached-fast-perl libjson-perl libclone-perl  libgraphviz-perl 
apt-get install libmime-lite-perl  # needs to be done after installing exim4
apt-get install libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl 
apt-get install libxml-feedpp-perl libmongodb-perl liburi-find-perl

Modules that could not be installed using apt-get: 

cpan
install URI::Escape::XS
install Image::OCR::Tesseract # needs imagemagick installed first

Modules that HAVE to be installed using cpan:

As of Feb 3rd 2015, the MongoDB perl module installed through "apt-get install libmongodb-perl" is too old (0.45): 

perl -MMongoDB -e 'print $MongoDB::VERSION . "\n"'
0.45

This causes an error when trying to access a list of "tags" such as categories: 

Software error:
Not an ARRAY reference at /home/stephane/product-opener/cgi/Blogs/Display.pm line 797.

A newer MongoDB perl module needs to be installed with cpan. 

cpan
install MongoDB

perl -MMongoDB -e 'print $MongoDB::VERSION . "\n"'
v0.708.0.0

cgi/startup.pl preloads a lot of modules, you can check that those modules are installed by running: 

cd /home/stephane/product-opener/cgi
perl startup.pl

Create some paths and links

cd /home/stephane/product-opener/cgi
ln -s ./ Blogs
ln -s S

Update IP to country GeoIP database

Product Opener uses the free legacy GeoIP Lite database from Maxmind:

The database installed with apt-get install geoip-database can be very old. To update manually, download the new databases to: 

/usr/share/GeoIP# ls -lrt
total 7256
-rw-r--r-- 1 root root 1103675  5 déc.   2013 GeoIPv6.dat
-rw-r--r-- 1 root root  600352  5 déc.   2013 GeoIP.dat


wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
gzip -d GeoIP.dat.gz
Retrieved from "https://wiki.openfoodfacts.org/index.php?title=Product_Opener/Installation/Debian_or_Ubuntu&oldid=4940"