Changes

Jump to navigation Jump to search
no edit summary
Line 1: Line 1:  +
[[Category:Infrastructure]]
 
= Cluster setup =
 
= Cluster setup =
   Line 15: Line 16:  
== Cluster networking ==
 
== Cluster networking ==
   −
At the networking level, a vRack links the cluster nodes with a 3Gbps private network used to access data on storage servers and replicate data between nodes. MTU is set to 9600 in the private network to take advantage of the high bandwidth.
+
At the networking level, a vRack links the cluster nodes with a 3Gbps private network used to access data on storage servers and replicate data between nodes. [https://docs.ovh.com/gb/en/dedicated/network-jumbo/ MTU is set to 9000] in the private network to take advantage of the high bandwidth.
    
== Cluster storage ==
 
== Cluster storage ==
Line 61: Line 62:     
Promox '''High-Availablity''' (HA) feature automatically migrates containers when the node hosting them is down. This is detected by the other nodes, forming a "quorum" (a majority of nodes considering a node is down in the cluster). In that case, the container is started with the last replicated storage. This means that the replication frequency of these HA nodes should be high (a few minutes) as changes since the last replication we be lost in the migration.
 
Promox '''High-Availablity''' (HA) feature automatically migrates containers when the node hosting them is down. This is detected by the other nodes, forming a "quorum" (a majority of nodes considering a node is down in the cluster). In that case, the container is started with the last replicated storage. This means that the replication frequency of these HA nodes should be high (a few minutes) as changes since the last replication we be lost in the migration.
 +
 +
=== Docker in a container ===
 +
'''Deprecated:''' We don't use this at it leads to weird bugs and there is no support for this configuration.
 +
 +
When Docker is run inside a container, it needs a few tweaks:
 +
 +
* first of all, install Docker from [https://docs.docker.com/engine/install/debian/#install-using-the-repository official repositories] and NOT from your Linux distribution
 +
* edit <code>/etc/pve/lxc/120.conf</code> (if the CT number is 120), and add: <code>features: keyctl=1,nesting=1</code>
 +
* reboot the container.
    
== Usage guidelines (to be completed) ==
 
== Usage guidelines (to be completed) ==
 
Here is a few guides to follow for all new virtual machines:
 
Here is a few guides to follow for all new virtual machines:
# MUST: no direct root access on the nodes, even with SSH key.
+
# MUST: no SSH root login on the nodes, even with SSH key.
 
# MUST: sudoers (root access using sudo) limited to SSH key based authentication
 
# MUST: sudoers (root access using sudo) limited to SSH key based authentication
 
# SHOULD: use SSH keys published on Github: giving access to a server is then simple and secure: <pre>curl https://github.com/CharlesNepote.keys | tee -a ~/.ssh/authorized_keys</pre>
 
# SHOULD: use SSH keys published on Github: giving access to a server is then simple and secure: <pre>curl https://github.com/CharlesNepote.keys | tee -a ~/.ssh/authorized_keys</pre>
 
# SHOULD: take care of production resources: use "nice" / "ionice" for scripts manually launched. Stéphane's tip: just use <pre>nice ./mycommand whatever arguments</pre> (nice default to lower the priority). CPU and I/O priorities can be set if needed at the virtualization level.
 
# SHOULD: take care of production resources: use "nice" / "ionice" for scripts manually launched. Stéphane's tip: just use <pre>nice ./mycommand whatever arguments</pre> (nice default to lower the priority). CPU and I/O priorities can be set if needed at the virtualization level.
 +
 +
= Usages =
 +
 +
== SSH connection ==
 +
 +
User accounts and SSH keys are directly created using data from github accounts.
 +
 +
At creation, the linux username matches the github account name, and the SSH keys are retrieved from your github public keys.
 +
 +
''(TODO: detect SSH key changes on github and inform administrator to update them if confirmed)''
 +
 +
=== If you have only one SSH key ===
 +
You can connect directly with:
 +
<pre>$ ssh -J [FQDN server] [internal IP]</pre>
 +
 +
Example:
 +
<pre>$ ssh -J ovh1.openfoodfacts.org 10.1.0.103</pre>
 +
 +
=== If you have many SSH keys ===
 +
<pre>ssh -i ~/.ssh/github_rsa -J stephanegigandet@ovh1.openfoodfacts.org stephanegigandet@10.1.0.103</pre>
 +
 +
You can modify your <code>~/.ssh/config</code> file and add:
 +
<pre>
 +
Host robotoff
 +
identityfile ~/.ssh/github_rsa
 +
proxyjump stephanegigandet@ovh1.openfoodfacts.org
 +
hostname 10.1.0.103
 +
user stephanegigandet
 +
</pre>
 +
Then you can connect yourself with a simpler command:
 +
<pre>$ ssh robotoff</pre>Tip to copy files from the host to your machine:
 +
$ scp -i ~/.ssh/id_rsa -J CharlesNepote@ovh1.openfoodfacts.org CharlesNepote@10.1.0.200:/home/CharlesNepote/file.csv ./myfile.csv
 +
 +
$ # or simpler, if you have created a new Host in your ~/.ssh/config file:
 +
 +
$ scp host_name:/home/CharlesNepote/file.csv ./myfile.csv
 +
 +
== Container creation ==
 +
 +
You need to have an account on the Proxmox infra.
 +
 +
* Login to the Proxmox web interface
 +
* Click on "Create CT"
 +
* Use a "Hostname" to let people know what it is about. Eg. "robotoff", "wiki", "proxy"...
 +
* Password: put something complex and forget it, as we will connect through SSH and not the web interface
 +
* Create a root password and add a ssh key.
 +
* Swap: 256
 +
* Network:
 +
** Bridge: vmbr0
 +
** IPv4: 10.1.0.104/24 (you need to use /24; end of IPv4 should be the same as the Proxmox container id; container 101 has the 10.1.0.101 IP).
 +
** Gateway: 10.0.0.1
 +
* Then connect to ovh1 or ovh2 machine, and launch the following scripts:
 +
** <code>$ sudo /root/cluster-scripts/ct_postinstall # choose the container ID when asked</code>
 +
** <code>$ sudo /root/cluster-scripts/mkuser</code> (if the GitHub username is the desired username)
 +
** or <code>$ sudo /root/cluster-scripts/mkuseralias</code> (to specify a different username than the GitHub username)
 +
* then you can login to the machine (see SSH connexion)
 +
* Also check "options" of the container and:
 +
** Start at boot: Yes
 +
** Protection: Yes (to avoid deleting it by mistake)
 +
* And add replication to ovh3:
 +
** In the Replication menu of the container, "Add" one
 +
** Target: ovh3
 +
** Schedule: */5 if you want every 5 minutes (takes less than 10 seconds, thanks to ZFS)
 +
 +
== Container administration ==
 +
 +
From ovh1, you can use the "pct" (Proxmox container) command:
 +
 +
* sudo pct list : list of containers
 +
* sudo pct enter 101 : enter container 101
 +
 +
== Reverse proxy setup to access a web server on a container ==
 +
 +
To access a web server running on a container (e.g. 111), requests have first to pass through a nginx reverse proxy running on container 101:
 +
 +
* create a CNAME record pointing the desired domain (e.g. metabase.openfoodfacts.org) to ovh1.openfoodfacts.org.
 +
* enter container 101 to setup the reverse proxy:
 +
** copy one of the existing config files (e.g. feedme.conf):
 +
*** root@proxy:/etc/nginx/conf.d# cp feedme.conf metabase.conf # don't forget the .conf at the end
 +
** edit the config file
 +
*** replace the domain name, directories, and target container (and possibly port)
 +
*** remove the lines that load the SSL certificate (as it does not exist yet)
 +
** create the SSL certificate
 +
*** root@proxy:/etc/nginx/conf.d# certbot --nginx
 +
 +
== VM creation (QEMU) ==
 +
 +
* Login to the Proxmox web interface
 +
* Click on "Create VM"
 +
* General:
 +
** USE A VM ID > 200; eg. 201
 +
** Use a "Hostname" to let people know what it is about. Eg. "robotoff", "wiki", "proxy"...
 +
* System: Leave all system settings to default
 +
* Hard Disk: select disk size (leave other settings to default)
 +
* CPU:
 +
** Type: host
 +
** Leave default for other settings
 +
* Memory: select the RAM you want
 +
* Network: leave all settings to default
 +
Then install OS with noVNC in the proxmox UI.
 +
 +
Tips for Debian install:
 +
 +
* IP: 10.1.0.201/8 for the installation and then switch to /24 when install is done
 +
* Gateway IP: 10.0.0.1/8 for the installation and then switch to 10.0.0.1/24 when install is done
 +
* Filesystem: ext4
 +
 +
= See also, inspirations =
 +
* [https://wiki.openstreetmap.org/wiki/FR:Serveurs_OpenStreetMap_France FR:Serveurs OpenStreetMap France] (french)
 +
* [https://wiki.openstreetmap.org/wiki/FR:Serveurs/Utilisation_des_serveurs FR:Serveurs/Utilisation des serveurs] (french)

Navigation menu