Difference between revisions of "Infrastructure"

From Open Food Facts wiki
Jump to navigation Jump to search
Line 108: Line 108:
 
apt-get install nginx
 
apt-get install nginx
  
 +
===== nginx configuration =====
 +
 +
<pre>
 +
/etc/nginx/sites-available# more off
 +
##
 +
# You should look at the following URL's in order to grasp a solid understanding
 +
# of Nginx configuration files in order to fully unleash the power of Nginx.
 +
# http://wiki.nginx.org/Pitfalls
 +
# http://wiki.nginx.org/QuickStart
 +
# http://wiki.nginx.org/Configuration
 +
#
 +
# Generally, you will want to move this file somewhere, and start with a clean
 +
# file but keep this around for reference. Or just disable in sites-enabled.
 +
#
 +
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
 +
##
 +
 +
# Default server configuration
 +
#
 +
server {
 +
        listen 80 default_server;
 +
        listen [::]:80 default_server;
 +
 +
        server_name openfoodfacts.eu *.openfoodfacts.eu;
 +
 +
        # SSL configuration
 +
        #
 +
        # listen 443 ssl default_server;
 +
        # listen [::]:443 ssl default_server;
 +
        #
 +
        # Self signed certs generated by the ssl-cert package
 +
        # Don't use them in a production server!
 +
        #
 +
        # include snippets/snakeoil.conf;
 +
 +
        root /home/off/html;
 +
 +
        # Add index.php to the list if you are using PHP
 +
        index index.html index.htm index.nginx-debian.html;
 +
 +
        location ~ ^/(images|js|rss|data|files|resources|foundation)/ {
 +
                # First attempt to serve request as file, then
 +
                # as directory, then fall back to displaying a 404.
 +
                try_files $uri $uri/ =404;
 +
        }
 +
 +
        location = /robots.txt {
 +
                try_files $uri $uri/ =404;
 +
        }
 +
 +
        location / {
 +
                proxy_set_header Host $host;
 +
                proxy_set_header X-Real-IP $remote_addr;
 +
                proxy_pass http://127.0.0.1:8001/cgi/display.pl?;
 +
        }
 +
 +
        location /cgi/ {
 +
                proxy_set_header Host $host;
 +
                proxy_set_header X-Real-IP $remote_addr;
 +
                proxy_pass http://127.0.0.1:8001;
 +
        }
 +
 +
        # deny access to .htaccess files, if Apache's document root
 +
        # concurs with nginx's one
 +
        #
 +
        #location ~ /\.ht {
 +
        #      deny all;
 +
        #}
 +
}
 +
</pre>
 +
 +
/etc/nginx/sites-enabled# ln -s /etc/nginx/sites-available/off off
 +
rm default
 +
 +
service nginx restart
 +
 +
To check for errors:
 +
 +
systemctl -l status nginx.service
 +
 +
===== Apache configuration =====
 +
 +
off.conf:
 +
 +
<pre>
 +
/etc/apache2/sites-available# cat off.conf
 +
# LoadModule perl_module modules/mod_perl.so
 +
 +
PerlWarn Off
 +
PerlRequire /home/off/cgi/startup.pl
 +
 +
 +
<Location /cgi>
 +
SetHandler perl-script
 +
PerlResponseHandler ModPerl::Registry
 +
PerlOptions +ParseHeaders
 +
Options +ExecCGI
 +
Require all granted
 +
</Location>
 +
 +
 +
<VirtualHost *>
 +
DocumentRoot /home/off/html
 +
ServerName openfoodfacts.eu
 +
ErrorLog /home/off/logs/error_log
 +
CustomLog /home/off/logs/access_log combined
 +
LogLevel debug
 +
ScriptAlias /cgi/ "/home/off/cgi/"
 +
 +
<Directory /home/off/html>
 +
Require all granted
 +
</Directory>
 +
 +
</VirtualHost>
 +
 +
PerlPostReadRequestHandler My::ProxyRemoteAddr
 +
 +
</pre>
 +
 +
 +
<pre>
 +
/etc/apache2/sites-enabled# ls -lrt
 +
total 0
 +
lrwxrwxrwx 1 root root 35 Jun 13 22:12 000-default.conf -> ../sites-available/000-default.conf
 +
/etc/apache2/sites-enabled# rm 000-default.conf
 +
/etc/apache2/sites-enabled# ln -s ../sites-available/off.conf off.conf
 +
</pre>
 +
 +
Port 8001
 +
 +
<pre>
 +
 +
/etc/apache2# vi ports.conf
 +
 +
#Listen 80
 +
Listen 8001
 +
 +
</pre>
 +
 +
service apache2 restart
 +
 +
To check for errors:
 +
 +
systemctl -l status apache2.service
 +
 +
mkdir /home/off/logs
  
 
==== Product Opener ====
 
==== Product Opener ====
Line 153: Line 299:
 
vi /home/cgi/Config2.pm
 
vi /home/cgi/Config2.pm
 
-> put right values for server domain, home path, and mongodb database name
 
-> put right values for server domain, home path, and mongodb database name
 +
</pre>
 +
 +
 +
===== robots.txt =====
 +
 +
Since we will run a copy of OFF on a separate domain, add a line to forbid robots completely.
 +
 +
<pre>
 +
/home/off/html# vi robots.txt
 +
User-agent: *
 +
Disallow: /
 +
Disallow: /cgi
 +
Disallow: /code
 +
~
 
</pre>
 
</pre>

Revision as of 12:51, 17 June 2016

Infrastructure

This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects.

New server install log

OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects. On June 13th 2016, a new dedicated server has been ordered specifically for OFF and OBF.

Hardware

Server setup

Server configuration

  • uname -a
    • Linux ns3362784.ip-37-187-74.eu 3.14.32-xxxx-grs-ipv6-64 #7 SMP Wed Jan 27 18:05:09 CET 2016 x86_64 GNU/Linux
  • perl -v
    • This is perl 5, version 20, subversion 2 (v5.20.2) built for x86_64-linux-gnu-thread-multi

Basic configuration

  • apt-get update
  • apt-get upgrade
  • apt-get install fail2ban
  • apt-get install sudo
  • apt-get install build-essential
  • apt-get install git


Users

  • admin users with sudo access
  • off user


DNS

Product Opener needs a domain, with a A record for the domain itself and another wildcard A record for all subdomains.

For testing the new server, we will be using openfoodfacts.eu

Product Opener dependencies

exim
  • apt-get install exim4
  • dpkg-reconfigure exim4-config
    • Internet Site mail is sent by smtp
    • 127.0.0.1
MongoDB

See https://docs.mongodb.com/manual/tutorial/install-mongodb-on-debian/

apt-get install mongodb

MongoDB shell version: 2.4.10

service mongod stop mv /var/lib/mongodb /home/mongodb

vi /etc/mongod.conf 

#  dbPath: /var/lib/mongodb
  dbPath: /home/mongodb

service mongod start

Stars with some warnings: 

mongo
MongoDB shell version: 3.2.7
connecting to: test
Server has startup warnings:
2016-06-13T19:34:08.245+0200 I CONTROL  [initandlisten]
2016-06-13T19:34:08.246+0200 I CONTROL  [initandlisten] ** WARNING: Cannot detect if NUMA interleaving is enabled. Failed to probe "/sys/devices/system/node/node1": Permission denied
2016-06-13T19:34:08.246+0200 W CONTROL  [initandlisten]
2016-06-13T19:34:08.246+0200 W CONTROL  [initandlisten] Failed to probe "/sys/kernel/mm/transparent_hugepage": Permission denied
2016-06-13T19:34:08.246+0200 W CONTROL  [initandlisten]
2016-06-13T19:34:08.246+0200 W CONTROL  [initandlisten] Failed to probe "/sys/kernel/mm/transparent_hugepage": Permission denied
2016-06-13T19:34:08.246+0200 I CONTROL  [initandlisten]


Apache / mod_perl and nginx

Apache 2 + mod_perl serve the dynamically generated HTML pages from Product Opener.

nginx is installed on port 80 as a reverse proxy. It serves the static files (images, JS, CSS etc.) and proxies the dynamic requests to the Apache server on another port.

apt-get install apache2

  1. stop apache in order to be able to install nginx (default port 80)

service stop apache2

apt-get install nginx

nginx configuration
/etc/nginx/sites-available# more off
##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# http://wiki.nginx.org/Pitfalls
# http://wiki.nginx.org/QuickStart
# http://wiki.nginx.org/Configuration
#
# Generally, you will want to move this file somewhere, and start with a clean
# file but keep this around for reference. Or just disable in sites-enabled.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##

# Default server configuration
#
server {
        listen 80 default_server;
        listen [::]:80 default_server;

        server_name openfoodfacts.eu *.openfoodfacts.eu;

        # SSL configuration
        #
        # listen 443 ssl default_server;
        # listen [::]:443 ssl default_server;
        #
        # Self signed certs generated by the ssl-cert package
        # Don't use them in a production server!
        #
        # include snippets/snakeoil.conf;

        root /home/off/html;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        location ~ ^/(images|js|rss|data|files|resources|foundation)/ {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }

        location = /robots.txt {
                try_files $uri $uri/ =404;
        }

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://127.0.0.1:8001/cgi/display.pl?;
        }

        location /cgi/ {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://127.0.0.1:8001;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #       deny all;
        #}
}

/etc/nginx/sites-enabled# ln -s /etc/nginx/sites-available/off off rm default

service nginx restart

To check for errors:

systemctl -l status nginx.service

Apache configuration

off.conf: 

/etc/apache2/sites-available# cat off.conf
# LoadModule perl_module modules/mod_perl.so

PerlWarn Off
PerlRequire /home/off/cgi/startup.pl


<Location /cgi>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
PerlOptions +ParseHeaders
Options +ExecCGI
Require all granted
</Location>


<VirtualHost *>
DocumentRoot /home/off/html
ServerName openfoodfacts.eu
ErrorLog /home/off/logs/error_log
CustomLog /home/off/logs/access_log combined
LogLevel debug
ScriptAlias /cgi/ "/home/off/cgi/"

<Directory /home/off/html>
Require all granted
</Directory>

</VirtualHost>

PerlPostReadRequestHandler My::ProxyRemoteAddr




/etc/apache2/sites-enabled# ls -lrt
total 0
lrwxrwxrwx 1 root root 35 Jun 13 22:12 000-default.conf -> ../sites-available/000-default.conf
/etc/apache2/sites-enabled# rm 000-default.conf
/etc/apache2/sites-enabled# ln -s ../sites-available/off.conf off.conf

Port 8001 


/etc/apache2# vi ports.conf

#Listen 80
Listen 8001

service apache2 restart

To check for errors:

systemctl -l status apache2.service

mkdir /home/off/logs

Product Opener

Libraries
  • apt-get install zlib1g-dev
Perl modules

apt-get install libwww-perl libimage-magick-perl libxml-encoding-perl libtext-unaccent-perl libmime-lite-perl libcache-memcached-fast-perl libjson-perl libclone-perl libgraphviz-perl libmime-lite-perl libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl libxml-feedpp-perl libmongodb-perl liburi-find-perl libxml-simple-perl

Some modules seem not to have Debian packages and must be built using CPAN: 

cpan
install URI::Escape::XS
install Encode::Punycode
install GraphViz2
install HTML::Defang
install Algorithm::CheckDigits
install Geo::IP
install Image::OCR::Tesseract
install DateTime::Format::Mail
install DateTime::Format::CLDR
install DateTime::Locale
Symbolic links in cgi directory =

ls -lrt |grep -- "->"

Make sure all links are pointing to the right path. 

/home/off/cgi# rm Blogs
/home/off/cgi# ln -s /home/off/cgi Blogs

ln -s SiteLang_off.pm SiteLang.pm

vi /home/off/cgi/startup.pl
# Needs to be configured
use lib "/home/off/cgi/";

vi /home/cgi/Config2.pm
-> put right values for server domain, home path, and mongodb database name


robots.txt

Since we will run a copy of OFF on a separate domain, add a line to forbid robots completely. 

/home/off/html# vi robots.txt
User-agent: *
Disallow: /
Disallow: /cgi
Disallow: /code
~
Retrieved from "https://wiki.openfoodfacts.org/index.php?title=Infrastructure&oldid=5031"