INSTALL WEB SERVERS FOR NAU MOODLE CLUSTER (MARIADB CLIENT, NGINX, PHP 7)

We have 2 Web servers; web1 and web2. The cluster database is separate, so we only need here PHP and NGINX

PREREQUISITES

$ sudo yum update

SETUP MARIADB REPO AND INSTALL MARIADB CLIENT

select your OS to get the latest MariaDB repo
https://downloads.mariadb.org/mariadb/repositories
you can rename a similar name to MariaDB
$ sudo vi /etc/yum.repos.d/MariaDB.repo
copy paste this (for 64 bit):

# MariaDB 10.2 CentOS repository list - created 2017-10-19 20:17 UTC
# http://downloads.mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.2/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

update yum
$ sudo yum update
you will need to test mariadb connectivity through mysql client
$ sudo yum install MariaDB-client

FIREWALL
Firewall commands
$ sudo systemctl status firewalld
$ sudo systemctl start firewalld
$ sudo systemctl enable firewalld
$ sudo systemctl stop firewalld

add service or port to to exclude from firewall
$ sudo firewall-cmd --permanent --add-service=http
$ sudo firewall-cmd --permanent --add-service=https
$ sudo firewall-cmd --permanent --add-service=mysql
$ sudo firewall-cmd --reload
$ sudo systemctl restart firewalld.service
$ sudo iptables -L

test to access mysql remotely using virtual ip
$ sudo mysql -h 10.10.4.17 -u root -p

MariaDB [(none)]> exit;


TUNE UP KERNEL

https://www.digitalocean.com/community/tutorials/how-to-optimize-nginx-configuration
First tune up Kernel
get the total number of CPU cores of the machine
$ grep processor /proc/cpuinfo | wc -l

2

let’s become root to change the kernel
$ sudo -s
see how much open files the kernel can retreive
$ ulimit -a
assume 1024 for 0.5 Gig (512 MB) RAM so for 12 Gig change it to 24576
$ ulimit -n 24576
check the limit
$ ulimit -Hn

24576


INSTALL PHP7.1

Let’s first install epel-release and web static repository, then update repository
$ sudo yum install epel-release
$ sudo rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
$ sudo yum update
$ sudo yum search php71w

if you see them, then install php7.1
$ sudo yum install php71w-bcmath php71w-cli mod_php71w php71w-common php71w-dba php71w-devel php71w-embedded php71w-enchant php71w-fpm php71w-gd php71w-imap php71w-interbase php71w-intl php71w-ldap php71w-mbstring php71w-mcrypt php71w-mysqlnd php71w-odbc php71w-opcache php71w-pdo php71w-pdo_dblib php71w-pear php71w-pecl-apcu php71w-pecl-apcu-devel php71w-pecl-geoip php71w-pecl-igbinary php71w-pecl-igbinary-devel php71w-pecl-imagick php71w-pecl-imagick-devel php71w-pecl-libsodium php71w-pecl-memcached php71w-pecl-mongodb php71w-pecl-redis php71w-pecl-xdebug php71w-pgsql php71w-phpdbg php71w-process php71w-pspell php71w-recode php71w-snmp php71w-soap php71w-tidy php71w-xml php71w-xmlrpc

check if php 7.1 installed
$ php -version

PHP 7.1.8 (cli) (built: Aug  9 2017 19:19:49) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.1.8, Copyright (c) 1999-2017, by Zend Technologies
    with Xdebug v2.5.5, Copyright (c) 2002-2017, by Derick Rethans

upload max filesize
$ sudo vi /etc/php.ini

upload_max_filesize = 4096M
memory_limit = 2048M
post_max_size = 128M
max_input_vars = 5000
max_execution_time = 0
cgi.fix_pathinfo=0
allow_url_include = On

SSH2 SOLUTION FOR PHP7 ON CENTOS 7

First make sure you have gcc, php71w-devel, libssh2, libssh2-devel, wget and unzip packages installed
$ sudo yum install gcc php71w-devel libssh2 libssh2-devel wget unzip

go to opt folder
$ cd /opt/

download php7 branch source of php ssh2
$ sudo wget https://github.com/Sean-Der/pecl-networking-ssh2/archive/php7.zip
unzip and enter the folder
$ sudo unzip php7.zip
$ cd pecl-networking-ssh2-php7/

prepare build environment for this extension using phpize which will create the needed file and folder structure and configure file
$ sudo phpize

Configuring for:
PHP Api Version: 20160303
Zend Module Api No: 20160303
Zend Extension Api No: 320160303

check the configuration
$ sudo ./configure

…….

now make the extension, you should see build complete.
$ sudo make

…..
…..
Build complete.
Don’t forget to run ‘make test’.

finally install the extension
$ sudo make install

Installing shared extensions: /usr/lib64/php/modules/

put the extension manually
$ sudo vi /etc/php.d/ssh2.ini

extension=ssh2.so

check php info, you’ll see the ssh2 module
$ php -i

ssh2

SSH2 support => enabled
extension version => 0.12+dev
libssh2 version => 1.4.3
banner => SSH-2.0-libssh2_1.4.3

or re-check php modules, you’ll see ssh2 module enabled
$ php -m


OPCACHE SETTINGS

We already have setup opcache, let’s do the opcache settings

$ sudo vi /etc/php.d/opcache.ini

opcache.enable_cli=1
opcache.memory_consumption=128
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=4000
opcache.revalidate_freq=60
opcache.fast_shutdown=1
opcache.consistency_checks=1

let’s install Opcache-Status, ocp.php and opcache-gui on web1 and web2
$ cd /usr/share/nginx/html
$ wget https://raw.github.com/rlerdorf/opcache-status/master/opcache.php
$ wget https://gist.github.com/ck-on/4959032/raw/0b871b345fd6cfcd6d2be030c1f33d1ad6a475cb/ocp.php
$ wget https://raw.github.com/amnuts/opcache-gui/master/index.php -O op.php

restart php-fpm and nginx
$ sudo systemctl restart php-fpm
$ sudo systemctl restart nginx

you’ll see now the Zen OPcache for php
$ php -v

PHP 7.1.10 (cli) (built: Sep 30 2017 10:30:04) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
with Zend OPcache v7.1.10, Copyright (c) 1999-2017, by Zend Technologies
with Xdebug v2.5.5, Copyright (c) 2002-2017, by Derick Rethans

In the future settings, you can check out cache like this

https://moodle.na.edu/opcache.php
https://moodle.na.edu/ocp.php
http://moodle.na.edu/op.php


INSTALL NGINX MANUALLY (PREFERRED)

let’s remove nginx completely if pre-installed
$ sudo systemctl stop nginx
$ sudo systemctl stop php-fpm
$ sudo yum remove nginx

install the development tools
$ sudo yum groupinstall "Development Tools”
$ sudo yum install net-tools yum-utils htop autoconf automake bind-utils wget curl unzip gcc-c++ pcre-devel zlib-devel openssl-devel libtool make nmap-netcat ntp pam-devel gd gd-devel perl-ExtUtils-Embed geoip-devel gperftools-devel

create the root folder first
$ sudo mkdir -p /usr/share/nginx/html

download latest version of nginx manually from http://nginx.org/download/
the purpose with source code is we want to compile ngx_cache_purge along together
$ cd /tmp/
$ sudo wget http://nginx.org/download/nginx-1.13.5.tar.gz
$ sudo wget http://labs.frickle.com/files/ngx_cache_purge-2.3.tar.gz

nginx-1.13.5 and ngx_cache_purge-2.3 will be created
$ sudo tar -xzf nginx-1.13.5.tar.gz
$ sudo tar -xzf ngx_cache_purge-2.3.tar.gz
$ cd nginx-1.13.5
$ sudo ./configure --sbin-path=/usr/sbin --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-debug --with-http_stub_status_module --with-http_flv_module --with-http_ssl_module --with-http_dav_module --add-module=/tmp/ngx_cache_purge-2.3 --with-http_geoip_module --with-http_image_filter_module --with-http_gunzip_module --with-threads --with-file-aio --with-http_sub_module --with-http_gzip_static_module --with-http_auth_request_module --with-http_random_index_module --with-http_secure_link_module --with-http_slice_module --with-http_degradation_module --with-http_perl_module --with-mail --with-mail_ssl_module --with-stream --with-stream_ssl_module --with-google_perftools_module --with-cpp_test_module

………..
………..
Configuration summary
  + using threads
  + using system PCRE library
  + using system OpenSSL library
  + using system zlib library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/sbin"
  nginx modules path: "/usr/local/nginx/modules"
  nginx configuration prefix: "/etc/nginx"
  nginx configuration file: "/etc/nginx/nginx.conf"
  nginx pid file: "/var/run/nginx.pid"
  nginx error log file: "/var/log/nginx/error.log"
  nginx http access log file: "/var/log/nginx/access.log"
  nginx http client request body temporary files: "/var/lib/nginx/body"
  nginx http proxy temporary files: "/var/lib/nginx/proxy"
  nginx http fastcgi temporary files: "/var/lib/nginx/fastcgi"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"

make root
$ sudo su
$ make && make install

now let’s check it out
$ nginx -V

nginx version: nginx/1.13.5
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) 
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --sbin-path=/usr/sbin --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-debug --with-http_stub_status_module --with-http_flv_module --with-http_ssl_module --with-http_dav_module --add-module=/tmp/ngx_cache_purge-2.3 --with-http_geoip_module --with-http_image_filter_module --with-http_gunzip_module --with-threads --with-file-aio --with-http_sub_module --with-http_gzip_static_module --with-http_auth_request_module --with-http_random_index_module --with-http_secure_link_module --with-http_slice_module --with-http_degradation_module --with-http_perl_module --with-mail --with-mail_ssl_module --with-stream --with-stream_ssl_module --with-google_perftools_module --with-cpp_test_module

create the system file so that you can use the start, stop, enable and status commands
Source: https://www.nginx.com/resources/wiki/start/topics/examples/systemd/

$ vi /lib/systemd/system/nginx.service

[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

create the required folders
$ mkdir -p /var/lib/nginx
$ mkdir -p /var/lib/nginx/body
$ mkdir -p /var/lib/nginx/fastcgi

check syntax
$ nginx -t

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

nginx commands
$ sudo systemctl enable nginx
$ sudo systemctl start nginx
$ sudo systemctl stop nginx
$ sudo systemctl restart nginx
$ sudo systemctl status nginx

let’s check the ngx_cache_purge module again
$ nginx -V 2>&1 | grep ngx_cache_purge -o

ngx_cache_purge

add nginx user and group
$ sudo adduser --system --no-create-home --user-group -s /sbin/nologin nginx

the nginx home page should work

Web1: http://10.10.4.21/
Web2: http://10.10.4.22/

Note: We’ll update the nginx configs later according to the Moodle Cluster


NGINX AND PHP-FPM CONFIGURATIONS

PHP-FPM SETTINGS
let’s update php-fpm config
$ sudo vi /etc/php-fpm.d/www.conf

user = nginx
group = nginx
#listen = 127.0.0.1:9000
listen = /var/run/php-fpm/php-fpm.sock 
listen.owner = nginx
listen.group = nginx
listen.mode = 0660
listen.allowed_clients = 127.0.0.1
pm = static
pm.max_children = 300
security.limit_extensions = .php .php3 .php4 .php5 .php7

NGINX TUNEUP

worker_connections tells our worker processes how many people can simultaneously be served by Nginx
The default value is 768; however, considering that every browser usually opens up at least 2 connections/server, this number can half.
$ sudo vi /etc/nginx/nginx.conf

events {
worker_connections 8192;
}
…..
#we have 2 cpu cores
worker_processes 2;

Note:
max_clients = worker_processes * worker_connections ==> 2 * 8192 = 16384
for moodle https://docs.moodle.org/29/en/Performance_recommendations
MaxClients = Total available memory * 80% / Max memory usage of apache process
Memory usage of apache process is usually 10MB but Moodle can easily use up to 100MB per process
max_clients = 12 x 1024 * 80% /100 = 98.304
So for moodle the concurrent connections is decreased from 8192 to 98
$ ps -ylC nginx –sort:rss

S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD
S     0  4370     1  0  80   0  2208 23891 sigsus ?        00:00:00 nginx
S   998  4372  4370  0  80   0  6824 24800 ep_pol ?        00:00:25 nginx
S   998  4371  4370  0  80   0  6828 24800 ep_pol ?        00:00:41 nginx

NGINX SETTINGS

$ sudo vi /etc/nginx/nginx.conf

user  nginx;
worker_processes  2;

error_log  /var/log/nginx/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

pid        /run/nginx.pid;


events {
    worker_connections  8192;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    tcp_nopush     on;
    keepalive_timeout 200;
    types_hash_max_size 2048;
    client_body_buffer_size 32K;
    client_header_buffer_size 8k;
    client_max_body_size 512m;
    large_client_header_buffers 8 64k;
    client_body_timeout 3000;
    client_header_timeout 3000;
    send_timeout 300;
    fastcgi_buffers 8 128k;
    fastcgi_buffer_size 128k;
    proxy_connect_timeout     600;
    proxy_send_timeout          600;
    proxy_read_timeout          600;
    server_names_hash_bucket_size 64;

    #gzip  on;

    server {
        listen       80;
        server_name  moodle.na.edu;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            root   /usr/share/nginx/html;
            try_files $uri $uri/ =404;
            index  index.php index.html index.htm;
        }

        error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }

        location ~ [^/]\.php(/|$) {
            root   /usr/share/nginx/html;
            fastcgi_split_path_info  ^(.+\.php)(/.+)$;
            fastcgi_index   index.php;
            fastcgi_pass    unix:/var/run/php-fpm/php-fpm.sock;
            include         fastcgi_params;
            fastcgi_param   PATH_INFO       $fastcgi_path_info;
            fastcgi_param   SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_read_timeout 300;

            if ($request_filename ~* ^.?/([^/]?)$)
            {
               set $filename $1;
            }

            if ($filename ~* ^.*?\.(eot)|(ttf)|(woff)$){
               add_header Access-Control-Allow-Origin *;
            }
        }

        location /dataroot/ {
                internal;
                alias /var/moodledata/; # ensure the path ends with /
        }

        location /cachedir/ {
                internal;
                alias /var/moodledata/cache/; # ensure the path ends with /
        }

        location /localcachedir/ {
                internal;
                alias /var/moodledata/localcache/; # ensure the path ends with /
        }

        location /tempdir/ {
                internal;
                alias /var/moodledata/temp/; # ensure the path ends with /
        }

        location /filedir/ {
                internal;
                alias /var/moodledata/filedir/; # ensure the path ends with /
        }

        location ~* \.(?:ttf|ttc|otf|eot|woff|font.css)$ {
            add_header "Access-Control-Allow-Origin" "*";
            expires 1M;
            access_log off;
            add_header Cache-Control "public";
        }

    }
}

check out nginx syntax
$ sudo nginx -t

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

restart nginx server
$ sudo systemctl restart nginx
$ sudo systemctl restart php-fpm
$ sudo vi /usr/share/nginx/html/info.php

should work!!!

http://10.10.4.21/info.php
http://10.10.4.22/info.php

CONFIGURE MOODLE

create the moodle data folder and give right permission
$ sudo mkdir /var/moodledata
$ sudo chmod 777 /var/moodledata/

After moving moodle files under /usr/share/nginx/html/ setup the config
$ sudo vi /usr/share/nginx/html/config.php

$CFG->dbhost    = '10.10.4.17’;// eg 'localhost' or 'db.isp.com' or IP
$CFG->dbname    = 'nau_moodle_05302014’;
$CFG->dbuser    = 'blablabla';   // your database username
$CFG->dbpass    = 'blablabla';   // your database password
$CFG->logouturl = 'http://portal.office.com';
//$CFG->wwwroot   = 'https://moodle.na.edu';//for production
$CFG->wwwroot   = '10.10.4.21';//for test
$CFG->dataroot  = '/var/moodledata’;

....

// enabled by Mehmet Sen
     $CFG->xsendfile = 'X-Accel-Redirect';     // Nginx {@see http://wiki.nginx.org/XSendfile}
// If your X-Sendfile implementation (usually Nginx) uses directory aliases specify them
// in the following array setting:
     $CFG->xsendfilealiases = array(
         '/dataroot/' => $CFG->dataroot,
         '/cachedir/' => '/var/moodledata/cache',    // for custom $CFG->cachedir locations
         '/localcachedir/' => '/var/moodledata/localcache',    // for custom $CFG->localcachedir locations
         '/tempdir/'  => '/var/moodledata/temp',     // for custom $CFG->tempdir locations
         '/filedir'   => '/var/moodledata/filedir',  // for custom $CFG->filedir locations
         '/filterdir' => '/var/moodledata/filter'  // for custom $CFG->filter locations
     );


$CFG->session_handler_class = '\core\session\memcached';
      $CFG->session_memcached_save_path = '10.10.4.29:11211';
      $CFG->session_memcached_prefix = 'memc.sess.key.';
      $CFG->session_memcached_acquire_lock_timeout = 120;
      $CFG->session_memcached_lock_expire = 7200;       // Ignored if PECL memcached is below version 2.2.0


CONFIGURE WEB SERVER SCRIPTS

On web1 and web2
$ sudo vi mon_con.sh

#!/bin/sh
watch -n 10 "netstat -tn 2>/dev/null|grep :80|awk '{print \$5}'|cut -d: -f1|sort|uniq -c|sort -nr|head”

Sometimes, you need to free the memory on web1 or web2
$ sudo su
$ sync; echo 3 > /proc/sys/vm/drop_caches


INSTALL UNOCONV

on Web1 and Web2

unoconv version not supported
if this test fails, it indicates a potential problem
The version of unoconv you have installed is not supported. Moodle’s assignment grading feature requires version 0.7 or higher.

source: https://docs.moodle.org/33/en/Universal_Office_Converter_(unoconv)
remove existing openoffice and libreoffice
$ sudo yum remove openoffice* libreoffice*

install openoffice and libreoffice
$ sudo yum install openoffice* libreoffice-pyuno
$ cd /opt
$ sudo git clone https://github.com/dagwieers/unoconv.git

copy (we chose this)
$ sudo cp unoconv/unoconv /usr/bin/
or link unoconv to /usr/bin
$ sudo ln -s unoconv/unoconv /usr/bin/unoconv

install and run a unoconv listener
$ sudo vi /etc/systemd/system/unoconv.service

[Unit]
Description=Unoconv listener for document conversions
Documentation=https://github.com/dagwieers/unoconv
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=simple
Environment="UNO_PATH=/usr/lib64/libreoffice/program"
ExecStart=/usr/bin/unoconv --listener

[Install]
WantedBy=multi-user.target

unoconv commands (run first 2)
$ sudo systemctl enable unoconv.service
$ sudo systemctl start unoconv.service
$ sudo systemctl stop unoconv.service
$ sudo systemctl restart unoconv.service
$ sudo systemctl status unoconv.service

if selinux is enabled (our selinux is already disabled)
$ sudo #setsebool -P httpd_execmem on

Advertisements

One thought on “INSTALL WEB SERVERS FOR NAU MOODLE CLUSTER (MARIADB CLIENT, NGINX, PHP 7)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s