Infrastructure

Infrastructure

This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects.

High level infrastructure design

1 dedicated OVH server running:

• 1 nginx reverse proxy to serve static files (installed with apt-get)
• 1 apache 2.4 + mod_perl (installed with apt-get)
  • running two separate instances on different ports, 1 for OFF and 1 for OBF
• 1 MongoDB

History

From 2012 to 2016, OFF and OBF have been running on part of an OVH server (also used for other purposes):

• cat /etc/debian_version -> 6.0.10 (squeeze)
• 1 apache 2.4 setup as reverse proxy to serve static files (manually built)
• 1 apache 1.3 + mod_perl for OFF (manually built)
• 1 apache 1.3 + mod_perl for OBF (manually built)
• MongoDB 2.4.12 (installed from mongodb provided packages)

New server install log

OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects. On June 13th 2016, a new dedicated server has been ordered specifically for OFF and OBF.

Hardware

• E3-SAT-3 Intel Xeon E3 1245v2 4 c / 8 t 3.4 GHz+ 32 Go 2x 2 To SATA3 Soft 16* 40.00€ HT (SOIT 48.00€ TTC)
• http://www.soyoustart.com/fr/offres/e3-sat-3.xml

Server setup

• Order server and pay.
• Login to manager https://eu.soyoustart.com/manager
• Install server
  • Debian 8.4 Stable Jessie

Server configuration

• uname -a
  • Linux ns3362784.ip-37-187-74.eu 3.14.32-xxxx-grs-ipv6-64 #7 SMP Wed Jan 27 18:05:09 CET 2016 x86_64 GNU/Linux
• perl -v
  • This is perl 5, version 20, subversion 2 (v5.20.2) built for x86_64-linux-gnu-thread-multi

Basic configuration

• apt-get update
• apt-get upgrade
• apt-get install fail2ban
• apt-get install sudo

• apt-get install build-essential
• apt-get install git

Users

• admin users with sudo access
• off user

IP failover

1 ip failover for each service so that we can easily switch servers

• OFF: 178.33.252.125
• OBF: 178.33.104.169

Add to /etc/network/interfaces :

post-up /sbin/ifconfig eth0:0 178.33.104.169 netmask 255.255.255.255 broadcast 178.33.104.169
post-down /sbin/ifconfig eth0:0 down

post-up /sbin/ifconfig eth0:1 178.33.252.125 netmask 255.255.255.255 broadcast 178.33.252.125
post-down /sbin/ifconfig eth0:1 down

/etc/init.d/networking restart

• [ ok ] Restarting networking (via systemctl): networking.service.

DNS

Product Opener needs a domain, with a A record for the domain itself and another wildcard A record for all subdomains.

For testing the new server, we will be using openfoodfacts.eu

Product Opener dependencies

exim

• apt-get install exim4
• dpkg-reconfigure exim4-config
  • Internet Site mail is sent by smtp
  • 127.0.0.1

MongoDB

See https://docs.mongodb.com/manual/tutorial/install-mongodb-on-debian/

apt-get install mongodb

MongoDB shell version: 2.4.10

service mongod stop mv /var/lib/mongodb /home/mongodb

vi /etc/mongod.conf

#  dbPath: /var/lib/mongodb
  dbPath: /home/mongodb

service mongod start

Stars with some warnings:

mongo
MongoDB shell version: 3.2.7
connecting to: test
Server has startup warnings:
2016-06-13T19:34:08.245+0200 I CONTROL  [initandlisten]
2016-06-13T19:34:08.246+0200 I CONTROL  [initandlisten] ** WARNING: Cannot detect if NUMA interleaving is enabled. Failed to probe "/sys/devices/system/node/node1": Permission denied
2016-06-13T19:34:08.246+0200 W CONTROL  [initandlisten]
2016-06-13T19:34:08.246+0200 W CONTROL  [initandlisten] Failed to probe "/sys/kernel/mm/transparent_hugepage": Permission denied
2016-06-13T19:34:08.246+0200 W CONTROL  [initandlisten]
2016-06-13T19:34:08.246+0200 W CONTROL  [initandlisten] Failed to probe "/sys/kernel/mm/transparent_hugepage": Permission denied
2016-06-13T19:34:08.246+0200 I CONTROL  [initandlisten]

Apache / mod_perl and nginx

Apache 2 + mod_perl serve the dynamically generated HTML pages from Product Opener.

nginx is installed on port 80 as a reverse proxy. It serves the static files (images, JS, CSS etc.) and proxies the dynamic requests to the Apache server on another port.

apt-get install apache2

1. stop apache in order to be able to install nginx (default port 80)

service stop apache2

apt-get install nginx

nginx configuration

/etc/nginx/sites-available# more off
##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# http://wiki.nginx.org/Pitfalls
# http://wiki.nginx.org/QuickStart
# http://wiki.nginx.org/Configuration
#
# Generally, you will want to move this file somewhere, and start with a clean
# file but keep this around for reference. Or just disable in sites-enabled.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##

# Default server configuration
#
server {
        listen 80 default_server;
        listen [::]:80 default_server;

        server_name openfoodfacts.eu *.openfoodfacts.eu;

        # SSL configuration
        #
        # listen 443 ssl default_server;
        # listen [::]:443 ssl default_server;
        #
        # Self signed certs generated by the ssl-cert package
        # Don't use them in a production server!
        #
        # include snippets/snakeoil.conf;

        root /home/off/html;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        location ~ ^/(images|js|rss|data|files|resources|foundation)/ {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }

        location = /robots.txt {
                try_files $uri $uri/ =404;
        }

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://127.0.0.1:8001/cgi/display.pl?;
        }

        location /cgi/ {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://127.0.0.1:8001;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #       deny all;
        #}
}

/etc/nginx/sites-enabled# ln -s /etc/nginx/sites-available/off off rm default

service nginx restart

To check for errors:

systemctl -l status nginx.service

Apache configuration

Set the user to off

vi /etc/apache2/envvars

#export APACHE_RUN_USER=www-data
export APACHE_RUN_USER=off
#export APACHE_RUN_GROUP=www-data
export APACHE_RUN_GROUP=off

off.conf:

/etc/apache2/sites-available# cat off.conf
# LoadModule perl_module modules/mod_perl.so

PerlWarn Off
PerlRequire /home/off/cgi/startup_apache2.pl


<Location /cgi>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
PerlOptions +ParseHeaders
Options +ExecCGI
Require all granted
</Location>


<VirtualHost *>
DocumentRoot /home/off/html
ServerName openfoodfacts.eu
ErrorLog /home/off/logs/error_log
CustomLog /home/off/logs/access_log combined
LogLevel debug
ScriptAlias /cgi/ "/home/off/cgi/"

<Directory /home/off/html>
Require all granted
</Directory>

</VirtualHost>

PerlPostReadRequestHandler My::ProxyRemoteAddr

/etc/apache2/sites-enabled# ls -lrt
total 0
lrwxrwxrwx 1 root root 35 Jun 13 22:12 000-default.conf -> ../sites-available/000-default.conf
/etc/apache2/sites-enabled# rm 000-default.conf
/etc/apache2/sites-enabled# ln -s ../sites-available/off.conf off.conf

Port 8001

/etc/apache2# vi ports.conf

#Listen 80
Listen 8001

service apache2 restart

To check for errors:

systemctl -l status apache2.service

mkdir /home/off/logs

Debugging:

If Apache does not start, check the apache2 logs (e.g. for apache2-obf) :

/var/log/apache2-obf# tail -f error.log
[Mon Jul 04 15:38:52.616689 2016] [perl:error] [pid 15197:tid 120373734786944] Can't locate /home/obf/cgi/startup_apache2.pl in @INC (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.20.2 /usr/local/share/perl/5.20.2 /usr/lib/x86_64-linux-gnu/perl5/5.20 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.20 /usr/share/perl/5.20 /usr/local/lib/site_perl . /etc/apache2-obf) at (eval 2) line 1.\n
[Mon Jul 04 15:38:52.616762 2016] [perl:error] [pid 15197:tid 120373734786944] Can't load Perl file: /home/obf/cgi/startup_apache2.pl for server ns3362784.ip-37-187-74.eu:0, exiting...

-> forgot to create startup_apache2.pl, going to copy the one from off and change the path in it.

Running a 2nd Apache instance for OBF

OFF and OBF share the same code (except some configuration in the Config.pm and SiteLang.pm modules), but the code is not made to run two separate servers. So we need two run two separate instances of Apache, one for OFF and one for OBF. This also allows to restart one but not the other, to run different versions of the code etc.

There is some support to run multiple Apache instances. Documentation is in /usr/share/doc/apache2/README.multiple-instances

sh /usr/share/doc/apache2/examples/setup-instance obf
Setting up /etc/apache2-obf ...
Setting up /etc/init.d/apache2-obf ...
Setting up symlinks: a2enmod-obf a2dismod-obf a2ensite-obf a2dissite-obf a2enconf-obf a2disconf-obf apache2ctl-obf
Setting up /etc/logrotate.d/apache2-obf and /var/log/apache2-obf ...

Configuration files for the new instance are in /etc/apache2-obf :

• Copy and edit obf.conf:
  • /etc/apache2-obf/sites-available# cp off.conf obf.conf
  • vi obf.conf
    • :%s/off/obf/g
    • ServerName openbeautyfacts.org
• /etc/apache2-obf/sites-enabled# ls -lrt off.conf
• lrwxrwxrwx 1 root root 27 Jun 17 14:45 off.conf -> ../sites-available/off.conf
• /etc/apache2-obf/sites-enabled# rm off.conf
• /etc/apache2-obf/sites-enabled# ln -s ../sites-available/obf.conf obf.conf

vi /etc/apache2-obf/envvars

#export APACHE_RUN_USER=www-data
export APACHE_RUN_USER=off
#export APACHE_RUN_GROUP=www-data
export APACHE_RUN_GROUP=off

Port 8002

/etc/apache2-obf# vi ports.conf

#Listen 80
Listen 8002

service apache2-obf start
Failed to start apache2-obf.service: Unit apache2-obf.service failed to load: No such file or directory.

Found solution in https://lists.debian.org/debian-apache/2016/03/msg00017.html

systemctl enable apache2-obf
Synchronizing state for apache2-obf.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d apache2-obf defaults
Executing /usr/sbin/update-rc.d apache2-obf enable
systemctl start apache2-obf

To check for errors:

systemctl -l status apache2-obf.service

mkdir /home/obf/logs

Product Opener

Libraries

• apt-get install zlib1g-dev

Perl modules

apt-get install libwww-perl libimage-magick-perl libxml-encoding-perl libtext-unaccent-perl libmime-lite-perl libcache-memcached-fast-perl libjson-perl libclone-perl libgraphviz-perl libmime-lite-perl libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl libxml-feedpp-perl libmongodb-perl liburi-find-perl libxml-simple-perl

Some modules seem not to have Debian packages and must be built using CPAN:

cpan
install URI::Escape::XS
install Encode::Punycode
install GraphViz2
install HTML::Defang
install Algorithm::CheckDigits
install Geo::IP
install Image::OCR::Tesseract
install DateTime::Format::Mail
install DateTime::Format::CLDR
install DateTime::Locale

Symbolic links in cgi directory =

ls -lrt |grep -- "->"

Make sure all links are pointing to the right path.

/home/off/cgi# rm Blogs
/home/off/cgi# ln -s /home/off/cgi Blogs

ln -s SiteLang_off.pm SiteLang.pm

vi /home/off/cgi/startup.pl
# Needs to be configured
use lib "/home/off/cgi/";

vi /home/cgi/Config2.pm
-> put right values for server domain, home path, and mongodb database name

robots.txt

Since we will run a copy of OFF on a separate domain, add a line to forbid robots completely.

/home/off/html# vi robots.txt
User-agent: *
Disallow: /
Disallow: /cgi
Disallow: /code
~

Data

• Copy data to new server
  • Populate Mongodb by running /home/off/cgi/update_all_products_from_dir_in_mongodb.pl
• Copy images
• Make sur permissions are correct
  • chown -R off deleted* html index ingredients lang invalid lists products users tmp

= Product opener debug

Once Apache starts:

/home/off/logs# tail -f error_log

[Fri Jun 17 14:51:34.123696 2016] [perl:error] [pid 18764:tid 117837813839616] [client 127.0.0.1:35749] Can't locate object method "remote_ip" via package "Apache2::Connection" at /home/off/cgi/startup.pl line 76.\n, referer: http://openfoodfacts.eu/

-> created startup_apache2.pl, loaded in off.conf