Line 1:
Line 1:
Also see [[Product Opener Installation]]
Also see [[Product Opener Installation]]
+
[[Category:ProductOpener]]
+
[[Category:Infrastructure]]
+
This page lists all the steps necessary to install Product Opener on Debian, including a development environment.
+
+
= Warning! Those instructions are outdated and kept for reference only =
+
+
Please see http://en.wiki.openfoodfacts.org/Infrastructure for up-to-date install instructions.
+
+
= Installation on Jessie =
+
+
We start from a clean Jessie install.
+
+
== Get the code and configure it ==
+
+
<pre>
+
apt-get install git
+
+
git clone <repo> /srv/openfoodfacts
+
mkdir /srv/openfoodfacts/users
+
mkdir /srv/openfoodfacts/products
+
chown -R www-data:www-data /srv/openfoodfacts
+
+
cd /srv/openfoodfacts/cgi
+
rm Blogs
+
ln -s . Blogs
+
ln -s SiteLang_off.pm SiteLang.pm
+
</pre>
+
+
Edit a few variables in Config2.pm:
+
<pre>
+
$server_domain = "yourdomain.com";
+
$www_root="/srv/openfoodfacts/html";
+
$data_root="/srv/openfoodfacts";
+
</pre>
+
+
Also in Config_off.pm:
+
<pre>
+
$domain = 'yourdomain.com';
+
$contact_email = 'webmaster@yourdomain.com';
+
$admin_email = 'webmaster@yourdomain.com';
+
</pre>
+
+
And in startup.pl:
+
<pre>
+
use lib "/srv/openfoodfacts/cgi/";
+
</pre>
+
+
== Install packaged prerequisites ==
+
+
<pre>
+
apt-get install apache2 build-essential geoip-bin geoip-database imagemagick libbarcode-zbar-perl libcache-memcached-fast-perl libclone-perl libcrypt-passwdmd5-perl libdatetime-format-mail-perl \
+
libencode-detect-perl libgeo-ip-perl libgraphics-color-perl libgraphviz-perl libhtml-defang-perl libimage-magick-perl libjson-perl libmime-lite-perl libmime-lite-perl libmongodb-perl libperl-dev \
+
libtest-nowarnings-perl libtext-unaccent-perl liburi-escape-xs-perl liburi-find-perl libwww-perl libxml-encoding-perl libxml-feedpp-perl memcached mongodb tesseract-ocr tesseract-ocr-fra zlib1g-dev
+
</pre>
+
+
== Prepare CPAN ==
+
If you never used CPAN before, you can automatically configure it:
+
<pre>
+
(echo y;echo o conf prerequisites_policy follow;echo o conf commit)|cpan
+
</pre>
+
+
== Install a few Perl modules from CPAN ==
+
+
<pre>
+
cpan Net::IDN::Punycode
+
cpan Encode::Punycode
+
cpan GraphViz2
+
cpan Image::OCR::Tesseract
+
cpan Algorithm::CheckDigits
+
cpan XML::Simple
+
</pre>
+
+
Note: on Debian Jessie, you might need to install the follwoing to be able to get cpan to install GraphViz2
+
<pre>
+
apt-get install libtest-version-perl
+
</pre>
+
+
== Rebuild Apache 2.2 and mod_perl ==
+
+
<pre>
+
cd /usr/local/src/
+
wget http://apache.crihan.fr/dist/httpd/httpd-2.2.31.tar.gz
+
tar xvfz httpd-2.2.31.tar.gz
+
cd httpd-2.2.31
+
./configure --with-mpm=prefork --prefix=/usr/local --enable-rewrite --enable-proxy --enable-proxy_http --enable-deflate --disable-userdir --enable-headers
+
make
+
make install
+
+
cd /usr/local/src/
+
wget http://mir2.ovh.net/ftp.apache.org/dist/perl/mod_perl-2.0.9.tar.gz
+
tar xvfz mod_perl-2.0.9.tar.gz
+
cd mod_perl-2.0.9
+
perl Makefile.PL MP_APXS=/usr/local/bin/apxs
+
make
+
make install
+
+
cat >> /usr/local/conf/httpd.conf <<EOF
+
Listen 8001
+
+
LoadModule perl_module modules/mod_perl.so
+
+
PerlWarn Off
+
PerlRequire /srv/openfoodfacts/cgi/startup.pl
+
+
<VirtualHost *:8001>
+
DocumentRoot /srv/openfoodfacts/html
+
ServerName yourdomain.com
+
ErrorLog /var/log/apache2/error.log
+
CustomLog /var/log/apache2/access.log combined
+
ScriptAlias /cgi/ /srv/openfoodfacts/cgi/
+
+
<Location /cgi>
+
SetHandler perl-script
+
PerlResponseHandler ModPerl::Registry
+
PerlOptions +ParseHeaders
+
Options +ExecCGI
+
Order allow,deny
+
Allow from all
+
</Location>
+
</VirtualHost>
+
+
PerlPostReadRequestHandler My::ProxyRemoteAddr
+
EOF
+
+
sed -i -e 's/^Listen 80$/#Listen 80/' /usr/local/conf/httpd.conf
+
sed -i -e 's/^User.*/User www-data/' /usr/local/conf/httpd.conf
+
sed -i -e 's/^Group.*/Group www-data/' /usr/local/conf/httpd.conf
+
+
/usr/local/bin/apachectl -k start
+
</pre>
+
+
== Configure standard Apache 2 ==
+
+
<pre>
+
a2enmod proxy_http
+
a2enmod rewrite
+
+
cat >> /etc/apache2/sites-available/product-opener-proxy.conf <<EOF
+
<VirtualHost *>
+
DocumentRoot /srv/openfoodfacts/html
+
ServerName yourdomain.com
+
ServerAlias *.yourdomain.com
+
+
<Directory "/srv/openfoodfacts/html">
+
Options Indexes FollowSymLinks
+
Require all granted
+
</Directory>
+
ProxyPreserveHost On
+
RewriteEngine on
+
RewriteCond %{REQUEST_URI} !/./
+
RewriteRule ^(/cgi/.*)$ http://localhost:8001$1 [P,L]
+
RewriteMap escape int:escape
+
RewriteRule ^/favicon.ico$ /favicon.ico [L]
+
RewriteCond %{REQUEST_URI} !^/images/
+
RewriteCond %{REQUEST_URI} !^/js/
+
RewriteCond %{REQUEST_URI} !^/foundation/
+
RewriteCond %{REQUEST_URI} !^/rss/
+
RewriteCond %{REQUEST_URI} !^/robots
+
RewriteCond %{REQUEST_URI} !^/clicks/
+
RewriteCond %{REQUEST_URI} !^/data/
+
RewriteCond %{REQUEST_URI} !^/files/
+
RewriteRule ^(.*)$ http://localhost:8001/cgi/display.pl?${escape:$1} [P,L,QSA]
+
</VirtualHost>
+
EOF
+
+
a2ensite product-opener-proxy.conf
+
a2dissite 000-default.conf
+
service apache2 restart
+
</pre>
+
+
You should now have a working Productopener instance running at http://yourdomain.com/. Enjoy!
+
+
= Installation on Debian Jessie with stock Apache and nginx =
+
+
<pre>
+
Vous êtes sur le point d'installer un OS sur le serveur ks3095298.kimsufi.com. Choisissez le type d'installation que vous souhaitez utiliser :
+
+
-> Installer à partir d'un template OVH
+
+
Debian 8.1 stable (Jessie) (stable) - debian8
+
+
root@ks3095298:~# uname -a
+
Linux ks3095298.kimsufi.com 3.14.32-xxxx-grs-ipv6-64 #5 SMP Wed Sep 9 17:24:34 CEST 2015 x86_64 GNU/Linux
+
+
perl -v
+
+
This is perl 5, version 20, subversion 2 (v5.20.2) built for x86_64-linux-gnu-thread-multi
+
−
This page lists all the steps necessary to install Product Opener on Debian, including a development environment.
+
apt-get update
+
apt-get upgrade
+
+
apt-get install fail2ban
+
apt-get install sudo
+
+
apt-get install mongodb
+
+
MongoDB shell version: 2.4.10
+
+
vi /etc/mongodb.conf
+
- change dbpath
+
- uncomment :
+
#port = 27017
+
#nohttpinterface = true
+
+
+
apt-get install git
+
+
apt-get install apache2
+
apt-get install libapache2-mod-perl2
+
apt-get install modperl
+
+
adduser off
+
+
apt-get install imagemagick
+
apt-get install memcached
+
apt-get install tesseract-ocr tesseract-ocr-fra
+
apt-get install exim4
+
dpkg-reconfigure exim4-config
+
apt-get install geoip-bin geoip-database
+
apt-get install zlib1g-dev
+
+
apt-get install libwww-perl libimage-magick-perl libxml-encoding-perl libtext-unaccent-perl libmime-lite-perl libcache-memcached-fast-perl libjson-perl libclone-perl libgraphviz-perl libmime-lite-perl libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl libxml-feedpp-perl libmongodb-perl liburi-find-perl libxml-simple-perl
+
+
+
cp -a /home/openfoodfacts-bitbucket/product-opener/cgi /home/off/cgi/
+
+
/home/off/cgi# rm Blogs
+
root@ks3095298:/home/off/cgi# ln -s /home/off/cgi Blogs
+
+
ln -s SiteLang_off.pm SiteLang.pm
+
+
vi /home/off/startup.pl
+
# Needs to be configured
+
use lib "/home/off/cgi/";
+
+
+
ln -s ../sites-available/001-off.conf 001-off.conf
+
service apache2 restart
+
+
To see errors, missing perl modules etc.
+
+
/var/log/apache2# tail -f error.log
+
+
cpan
+
install URI::Escape::XS
+
install Encode::Punycode
+
install GraphViz2
+
install HTML::Defang
+
install Algorithm::CheckDigits
+
install Geo::IP
+
install Image::OCR::Tesseract
+
DateTime::Format::Mail
+
DateTime::Format::CLDR
+
DateTime::Locale
+
+
+
+
+
Issues:
+
+
MongoDB does not start:
+
+
service mongodb start
+
root@ks3095298:/home/off/cgi# ps -ef | grep mongo
+
root 21814 23471 0 23:56 pts/0 00:00:00 grep mongo
+
+
+
in /var/log/syslog:
+
+
Jan 11 00:05:05 ks3095298 systemd[1]: Starting An object/document-oriented database...
+
Jan 11 00:05:05 ks3095298 systemd[1]: Started An object/document-oriented database.
+
Jan 11 00:05:05 ks3095298 mongod[22404]: Mon Jan 11 00:05:05.434 terminate() called, printing stack (if implemented for platform):
+
Jan 11 00:05:05 ks3095298 mongod[22404]: 0xa6d3c6 0x57e204 0x7212186b7bb6 0x7212186b7c01 0x7212186b7e19 0x7212192b666d 0xa6070d 0xa61a67 0xa5f8ca 0xa5f979 0x5a626c 0x5a65d2 0x5a6a93 0x5a6ab9 0x5a6e5c 0x56647b 0x721217dbab45 0x57e05c
+
Jan 11 00:05:05 ks3095298 mongod[22404]: /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xa6d3c6]
+
+
+
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com --recv EA312927
+
+
+
Text::Unaccent was causing Apache to segfault, replaced the module with pure perl
+
+
remote_ip has been replaced in Apache 2.4, need to check if we do get the X-Forwarder-For ip directly now
+
+
+
Loading products from /products in the mongodb database:
+
+
update_all_products_from_dir_in_mongodb.pl
+
+
+
+
Trying to use nginx for the reverse proxy:
+
+
apt-get install nginx (need port 80 to be available, stop apache or change its port first)
+
+
+
+
root@ks3095298:/etc/nginx/sites-enabled# more /etc/nginx/sites-available/off
+
##
+
# You should look at the following URL's in order to grasp a solid understanding
+
# of Nginx configuration files in order to fully unleash the power of Nginx.
+
# http://wiki.nginx.org/Pitfalls
+
# http://wiki.nginx.org/QuickStart
+
# http://wiki.nginx.org/Configuration
+
#
+
# Generally, you will want to move this file somewhere, and start with a clean
+
# file but keep this around for reference. Or just disable in sites-enabled.
+
#
+
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
+
##
+
+
# Default server configuration
+
#
+
server {
+
listen 80 default_server;
+
listen [::]:80 default_server;
+
+
server_name openfoodfacts.ovh *.openfoodfacts.ovh;
+
+
# SSL configuration
+
#
+
# listen 443 ssl default_server;
+
# listen [::]:443 ssl default_server;
+
#
+
# Self signed certs generated by the ssl-cert package
+
# Don't use them in a production server!
+
#
+
# include snippets/snakeoil.conf;
+
+
root /home/off/html;
+
+
# Add index.php to the list if you are using PHP
+
index index.html index.htm index.nginx-debian.html;
+
+
location ~ ^/(images|js|rss|data|files|resources|foundation)/ {
+
# First attempt to serve request as file, then
+
# as directory, then fall back to displaying a 404.
+
try_files $uri $uri/ =404;
+
}
+
+
location = /robots.txt {
+
try_files $uri $uri/ =404;
+
}
+
+
location / {
+
proxy_set_header Host $host;
+
proxy_set_header X-Real-IP $remote_addr;
+
proxy_pass http://127.0.0.1:8001/cgi/display.pl?;
+
}
+
+
location /cgi/ {
+
proxy_set_header Host $host;
+
proxy_set_header X-Real-IP $remote_addr;
+
proxy_pass http://127.0.0.1:8001;
+
}
+
+
# deny access to .htaccess files, if Apache's document root
+
# concurs with nginx's one
+
#
+
#location ~ /\.ht {
+
# deny all;
+
#}
+
}
+
+
+
</pre>
+
+
= Old notes =
All instructions below are for user "stephane", please use your own name. :-)
All instructions below are for user "stephane", please use your own name. :-)
Line 33:
Line 399:
Create 2 A records in the DNS zone of the domain you want to use:
Create 2 A records in the DNS zone of the domain you want to use:
* openfoodfacts.ovh 94.23.195.82
* openfoodfacts.ovh 94.23.195.82
−
** *.openfoodfacts.ovh 94.23.195.82
+
* *.openfoodfacts.ovh 94.23.195.82
== Install ==
== Install ==
Line 120:
Line 486:
make
make
make install
make install
+
cd ~
+
cd ~/src
wget http://mir2.ovh.net/ftp.apache.org/dist/perl/mod_perl-2.0.8.tar.gz
wget http://mir2.ovh.net/ftp.apache.org/dist/perl/mod_perl-2.0.8.tar.gz
tar xvfz mod_perl-2.0.8.tar.gz
tar xvfz mod_perl-2.0.8.tar.gz
Line 127:
Line 495:
make
make
make install
make install
+
cd ~
</pre>
</pre>
==== Configuration ====
==== Configuration ====
−
In apache/conf/httpd.conf:
+
In ~/apache/conf/httpd.conf:
Change the port to something else than 80 (the reverse proxy will listen on 80 and forward to the backend on another port).
Change the port to something else than 80 (the reverse proxy will listen on 80 and forward to the backend on another port).
Line 177:
Line 546:
PerlPostReadRequestHandler My::ProxyRemoteAddr
PerlPostReadRequestHandler My::ProxyRemoteAddr
</pre>
</pre>
−
=== Apache 2.4 - light weight reverse proxy ===
=== Apache 2.4 - light weight reverse proxy ===
Line 196:
Line 564:
tar xvfz apr-1.5.1.tar.gz
tar xvfz apr-1.5.1.tar.gz
tar xvfz apr-util-1.5.4.tar.gz
tar xvfz apr-util-1.5.4.tar.gz
−
ln -s apr-1.5.1 ap
+
ln -s apr-1.5.1 apr
ln -s apr-util-1.5.4 apr-util
ln -s apr-util-1.5.4 apr-util
cd ..
cd ..
Line 203:
Line 571:
make
make
−
make install
+
sudo make install
+
cd ~
</pre>
</pre>
==== Configuration ====
==== Configuration ====
−
in proxy/conf/httpd.conf :
+
in ~/proxy/conf/httpd.conf :
* uncomment #LoadModule rewrite_module modules/mod_rewrite.so
* uncomment #LoadModule rewrite_module modules/mod_rewrite.so
* uncomment #LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
* uncomment #LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
Line 223:
Line 592:
CustomLog /home/stephane/logs/proxy_access_log combined
CustomLog /home/stephane/logs/proxy_access_log combined
ServerAdmin stephane@openfoodfacts.org
ServerAdmin stephane@openfoodfacts.org
+
+
<Location "/foundation/foundation-icons" >
+
Header set Access-Control-Allow-Origin "*"
+
</Location>
+
+
<Location "/images/products" >
+
Header set Link "<http://creativecommons.org/licenses/by-sa/3.0/>; rel='license'; title='CC-BY-SA 3.0'"
+
</Location>
+
<Directory "/home/stephane/product-opener/html">
<Directory "/home/stephane/product-opener/html">
Line 241:
Line 619:
RewriteCond %{REQUEST_URI} !^/data/
RewriteCond %{REQUEST_URI} !^/data/
RewriteCond %{REQUEST_URI} !^/files/
RewriteCond %{REQUEST_URI} !^/files/
+
RewriteCond %{REQUEST_URI} !^/resources/
+
RewriteCond %{REQUEST_URI} !^/foundation/
RewriteRule ^(.*)$ http://localhost:8001/cgi/display.pl?${escape:$1} [P,L,QSA]
RewriteRule ^(.*)$ http://localhost:8001/cgi/display.pl?${escape:$1} [P,L,QSA]
</VirtualHost>
</VirtualHost>
Line 262:
Line 642:
</pre>
</pre>
−
−
=== Install Perl modules ===
=== Install Perl modules ===
Line 274:
Line 652:
apt-get install libcache-memcached-fast-perl libjson-perl libclone-perl libgraphviz-perl
apt-get install libcache-memcached-fast-perl libjson-perl libclone-perl libgraphviz-perl
apt-get install libmime-lite-perl # needs to be done after installing exim4
apt-get install libmime-lite-perl # needs to be done after installing exim4
−
apt-get install libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl libmongodb-perl liburi-find-perl
+
apt-get install libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl
+
apt-get install libxml-feedpp-perl libmongodb-perl liburi-find-perl
</pre>
</pre>
Line 285:
Line 664:
</pre>
</pre>
+
Modules that HAVE to be installed using cpan:
+
+
As of Feb 3rd 2015, the MongoDB perl module installed through "apt-get install libmongodb-perl" is too old (0.45):
+
+
<pre>
+
perl -MMongoDB -e 'print $MongoDB::VERSION . "\n"'
+
0.45
+
</pre>
+
+
This causes an error when trying to access a list of "tags" such as categories:
+
+
<pre>
+
Software error:
+
Not an ARRAY reference at /home/stephane/product-opener/cgi/Blogs/Display.pm line 797.
+
</pre>
+
+
A newer MongoDB perl module needs to be installed with cpan.
+
+
<pre>
+
cpan
+
install MongoDB
+
</pre>
+
+
<pre>
+
perl -MMongoDB -e 'print $MongoDB::VERSION . "\n"'
+
v0.708.0.0
+
</pre>
+
+
cgi/startup.pl preloads a lot of modules, you can check that those modules are installed by running:
+
+
<pre>
+
cd /home/stephane/product-opener/cgi
+
perl startup.pl
+
</pre>
=== Create some paths and links ===
=== Create some paths and links ===
Line 292:
Line 705:
ln -s ./ Blogs
ln -s ./ Blogs
ln -s S
ln -s S
+
</pre>
+
+
=== Update IP to country GeoIP database ===
+
+
Product Opener uses the free legacy GeoIP Lite database from Maxmind:
+
* http://search.cpan.org/~maxmind/Geo-IP-1.45/lib/Geo/IP.pm
+
* http://dev.maxmind.com/geoip/legacy/geolite/
+
+
The database installed with apt-get install geoip-database can be very old.
+
To update manually, download the new databases to:
+
+
<pre>
+
/usr/share/GeoIP# ls -lrt
+
total 7256
+
-rw-r--r-- 1 root root 1103675 5 déc. 2013 GeoIPv6.dat
+
-rw-r--r-- 1 root root 600352 5 déc. 2013 GeoIP.dat
+
+
+
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
+
gzip -d GeoIP.dat.gz
+
</pre>