29
edits
Changes
Jump to navigation
Jump to search
Line 1:
Line 1: − + + + + + + + + + + + + + + + + + + + + + + + + + + + − The cluster is made of [[Infrastructure|4 physical machines]]: ovh1, ovh2, ovh3, ovh4. − + − + − + − +
Infrastructure/VM cluster (view source)
Revision as of 18:18, 13 January 2021
, 18:18, 13 January 2021Some general explanations about the cluster setup and features.
Open Food Facts uses a cluster to host different virtual machines (VM).
Open Food Facts uses a Proxmox based cluster to host different virtual machines (VM) on OVH provided servers.
The cluster is made of [[Infrastructure|4 physical machines]] ("nodes" or "hosts" in Proxmox jargon):
* ovh1 and ovh2 are '''computation''' oriented nodes: '''24 cores, 256 GB RAM''', 1TB nvme SSD
* ovh3 and ovh4 are '''storage''' oriented nodes: 32GB RAM, '''6x12 TB HDD''' + 512GB NVMe cache
ovh1 and ovh3 are in Roubaix datacenter, ovh2 and ovh4 in Strasbourg.
At initial setup (january 2021), v6.3 of proxmox has been installed (based on Debian 10 "buster").
Proxmox '''GUI''' is available on any of the cluster nodes on port 8006.
At the networking level, a vRack links the cluster nodes with a 3Gbps private network used to access data on storage servers and replicate data between nodes.
All storage is managed using '''ZFS''' which provides:
* volume management (like lvm)
* redundancy (like mdadm)
* encryption (like luks)
* compression
* snapshots
* quota
Snapshots allow efficient synchronization between remote storage, and is used extensively by Proxmox to replicate data across the nodes. Snapshots simplify backups and allow rollbacks.
Proxmox allows full virtualization (VM, using QEMU) are containerization (CT using LXC). For performance reason, LXC CTs are prefered over QEMU VMs.
All resources are shared and dynamically allocated, thus can be reallocated at any time with reboot.
== Usage guidelines (to be completed) ==
== Usage guidelines (to be completed) ==
Here is a few guides to follow for all new virtual machines:
Here is a few guides to follow for all new virtual machines:
# MUST: no direct root access without SSH key
# MUST: no direct root access on the nodes, even with SSH key.
# MUST: indirect root access with sudo, only with SSH connexion
# MUST: sudoers (root access using sudo) limited to SSH key based authentication
# SHOULD: use SSH keys published on Github: giving access to a server is then simple and secure: <pre>curl https://github.com/CharlesNepote.keys | tee -a /home/charles/.ssh/authorized_keys</pre>
# SHOULD: use SSH keys published on Github: giving access to a server is then simple and secure: <pre>curl https://github.com/CharlesNepote.keys | tee -a ~/.ssh/authorized_keys</pre>
# SHOULD: take care of production resources: use "nice" for scripts manually launched. Stéphane's tip: just use <pre>nice ./mycommand -arg1 foo -arg2 bar</pre> (nice default to lower the priority).
# SHOULD: take care of production resources: use "nice" / "ionice" for scripts manually launched. Stéphane's tip: just use <pre>nice ./mycommand whatever arguments</pre> (nice default to lower the priority). CPU and I/O priorities can be set if needed at the virtualization level.