Categories
Applications AWS EC2 Firewall Linux NethServer OpenVPN Roundcube Security/Vulnerability

NethServer on AWS

If you ever wanted to setup a VPN server, mail server, mail box, web server, or nextcloud (an open source google drive alternative), then you can go through this blog and set it up in minutes using NethServer.

NethServer new management web UI at 9090 port

Since AWS does not provide a NethServer AMI by default via MarketPlace or Community AMI, I had to make a VM image in my local server, modify it, import it to AWS and then modify it again and make it working for AWS.

I have made that image public and it is available in the Mumbai region. If you need it in any other region, contact me or you can make an AMI after launching the server in the Mumbai region and transfer it to whichever region you like.

So to setup a NethServer in AWS, below are the steps you have to follow

Setting up EC2

The instance

Go to the AWS EC2 console and click on Launch Instance

There go to Community AMIs and search for NethServer or you can search the below AMI ID. Do note that this AMI ID might change if I update the AMI

ami-0c0dc2c9b42edfa60

Security group rules

The security group should allow the following as per requirement for accessing the server

For the latest management web UI, open the port 9090 inbound

For old management web UI, open port 980 inbound

For SSH Access, open port 22 inbound

You can keep outbound access as full open for all traffic.

User Access

The NethServer AMI by default comes with a user. The details are as follows

Username: admin
Password: Nethserver@123

Domain/Other details

By default, this AMI comes with the FQDN ami.vigneshn.in

This can be changed and there are also my contact details in the Company name space so you can change it according to your requirements.

Also note that you have to change the FQDN before you install LDAP in account providers (Users and Groups).

You can also use Active directory (external) for user and group management.

Also you need to setup LDAP to change the password of the default admin user

Known caveats

Currently there is an issue with Nethserver where it requrires a green interface at any cost, without which it will throw and error at ipconf step.

To fix this, create a network interface in AWS and attach it to the instance.

Make this a green network and if your ip is 172.20.20.20 for the network interface, then give that as static in for green and the subnet mask as 255.255.255.0 and the gateway as 172.20.20.1

Documentation

For more details on how to set up the individual services in NethServer, you can visit the NethServer documentation here.

Categories
Amazon Linux 2 AWS Containerization EC2 ECS Issues Linux Outage Serverless WordPress

Why there was an outage for my site recently

So initially I was running this site in an EC2 instance which costed around 1200Rs. Then I learnt about serverless deployments and made my site serverless about which you can read here.

So after going serverless, I was receiving a monthly bill of 2400Rs which was fine since I could brag about my site being infinitely scalable.

But this month (August-2020) I was looking at my billing console and got a bill of 2200Rs by august 9th. This was alarming. The issue was the google search console was sending more and more requests to my site.

This woke up my Aurora serverless RDS from suspended mode to 1 ACU, and Aurora serverless is costly.

So I switched off the ECS from 22:00 to 10:00 using a lambda function but even then I was paying for the load balancer.

So now I have changed it back to an t3a.micro instance which costs me 400Rs per month.

This is why there was an outage.

Categories
Applications AWS CentOS Linux Virtual Machine Manager Virtualization

Setting up a CentOS 8 server as a virtualization host

If you are trying to setup virtual machines in a CentOS 8 server, the following are the steps.

I set this up in AWS with an m5d.metal instance (only metal instances allow direct hardware access in AWS. AWS does not support nested virtualization) but it is the same for any CentOS 8 server.

Once you get the server up and running, check if your hardware supports virtualization. If it is a new machine with a recent processor, it should support it but just check it just in case.

grep -E '(vmx|svm)' /proc/cpuinfo
lsmod | grep -i kvm

Now we install the X-Org server so that you can access the virt-manager GUI in the server on your local machine.

sudo yum install xorg-x11-xauth xorg-x11-fonts-* xorg-x11-utils

Now install all the required packages required for virtualization

sudo yum groupinstall "Virtualization Host"

Once the installation is complete, check the status of libvirtd service which is required to run the virtual machine.

sudo systemctl status libvirtd

If it is not running, then run the following command to start the service

sudo systemctl start libvirtd

Remember to enable the service so that the service starts up automatically after reboot

sudo systemctl enable libvirtd

In the current situation, only root can connect to the libvirtd daemon to create VMs but since we are planning to use the GUI and not the virt-install command line tool, it will be better to add the user to the libvirt group. This is because root doesn’t play will with X-Org server.

sudo usermod -aG libvirt centos

Now you have to log off and log in.

You can run the following command to kill all the process by the user (here it is centos) which will basically log you out.

sudo pkill -U centos

You can log back in with the -X option so that when you run the virt-manager, you can see it in your local machine.

ssh -i key.pem -X centos@IP

Once you have logged in, check if you are there in the libvirt group by running the following command.

groups

Confirm that the installation of packages was completed successfull by running the following command.

sudo virsh version

Once all that is done, you can run the following command to start the GUI Virtual machine manager.

virt-manager

That’s it. You may now setup virtual machines in the server using the same steps you have been following till now with Virtual machine manager.

Categories
Applications Cluster Linux MySQL

MySQL-Cluster group replication

Follow this to setup a MySQL database cluster with group replication. Group replication is where the database across the cluster is constantly synced, so you can write or read from any database and it will be replicated across the other ones.

This increases the redundancy and read throughput.

Assumptions

Following are the assumptions that are made.

Hostname of service DB server 1: service-db1
Hostname of service DB server 2: service-db2
Hostname of service DB server 3: service-db3

IP of service DB server 1: 172.32.0.10
IP of service DB server 2: 172.32.0.20
IP of service DB server 3: 172.32.0.30

MySQL port: 3306
MySQL cluster port: 33061

Install MySQL on all three servers

Add the MySQL community repo

Download the deb file

curl -OL https://repo.mysql.com//mysql-apt-config_0.8.15-1_all.deb

Install using dpkg

sudo dpkg -i mysql-apt-config_0.8.15-1_all.deb

Select mysql-5.7

Update the apt repo

sudo apt update

Install MySQL server

sudo apt install mysql-server

Start MySQL service

sudo systemctl start mysql && sudo systemctl enable mysql

Do secure installation

sudo mysql_secure_installation

Open configuration

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

Edit configuration

OptionOld valueNew value
bind-address127.0.0.10.0.0.0

Add /etc/hosts entries to make hostnames resolvable

Open hosts file

sudo vim /etc/hosts

Add the following configuration after changing the IPs appropriately

172.32.0.10 service-db1
172.32.0.20 service-db2
172.32.0.30 service-db3

Add firewall configuration

sudo ufw allow from 172.32.0.54 to any port 3306 proto tcp
sudo ufw allow from 172.32.0.175 to any port 3306 proto tcp
sudo ufw allow from 172.32.0.166 to any port 3306 proto tcp

sudo ufw allow out to 172.32.0.54 port 3306 proto tcp
sudo ufw allow out to 172.32.0.175 port 3306 proto tcp
sudo ufw allow out to 172.32.0.166 port 3306 proto tcp

sudo ufw allow out to 172.32.0.54 port 33061 proto tcp
sudo ufw allow out to 172.32.0.175 port 33061 proto tcp
sudo ufw allow out to 172.32.0.166 port 33061 proto tcp

sudo ufw allow from 172.32.0.54 to any port 33061 proto tcp
sudo ufw allow from 172.32.0.175 to any port 33061 proto tcp
sudo ufw allow from 172.32.0.166 to any port 33061 proto tcp

Prepare first MySQL server for group replication

Edit configuration

Open the file

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

Add the following content

disabled_storage_engines=”MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY”
server_id=1
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=binlog
binlog_format=ROW
plugin_load_add=’group_replication.so’
transaction_write_set_extraction=XXHASH64
group_replication_group_name=”40d2393e-a59b-11ea-8acd-0242ac110003″
group_replication_start_on_boot=off
group_replication_local_address= “service-db1:33061″
group_replication_group_seeds= “service-db1:33061,service-db2:33061,service-db3:33061″
group_replication_bootstrap_group=off
group_replication_ip_whitelist=”service-db1,service-db2,service-db3”

The server_id changes to 2 and 3 and group_replication_local_address changes to the hostname and make it resolvable using /etc/hosts file of the current instance for all the servers in mysql cluster, and also change the whitelist IP range.

Login to mysql

sudo mysql -u root

Get UUID

SELECT UUID();

Replace UUID in the configuration file

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

The UUID to be changed is at group_replication_group_name section

Restart MySQL server

sudo systemctl restart mysql

Login to mysql

sudo mysql -u root

Run the following commands

Turn off bin log

SET SQL_LOG_BIN=0;

Create replication user

CREATE USER rpl_user@’%’ IDENTIFIED BY ‘123456789’;

Give appropriate permission

GRANT REPLICATION SLAVE ON *.* TO rpl_user@’%’;

Flush permissions

FLUSH PRIVILEGES;

Turn on bin log

SET SQL_LOG_BIN=1;

Enable user for replication

CHANGE MASTER TO MASTER_USER=’rpl_user’, MASTER_PASSWORD=’123456789′ FOR CHANNEL ‘group_replication_recovery’;

Bootstrap the replication group

SET GLOBAL group_replication_bootstrap_group=ON;

Start the replication

START GROUP_REPLICATION;

Stop the bootstrap

SET GLOBAL group_replication_bootstrap_group=OFF;

Check if the node is online

SELECT * FROM performance_schema.replication_group_members;

Change the mysqld.cnf file group_replication_start_on_boot to on

Cluster second MySQL server to the group

Edit configuration

Open the file

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

Add the following content

disabled_storage_engines=”MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY”
server_id=2
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=binlog
binlog_format=ROW
plugin_load_add=’group_replication.so’
transaction_write_set_extraction=XXHASH64
group_replication_group_name=”40d2393e-a59b-11ea-8acd-0242ac110003″
group_replication_start_on_boot=off
group_replication_local_address= “service-db2:33061″
group_replication_group_seeds= “service-db1:33061,service-db2:33061,service-db3:33061″
group_replication_bootstrap_group=off
group_replication_ip_whitelist=”service-db1,service-db2,service-db3”

The server_id changes to 2 and 3 and group_replication_local_address changes to the hostname and make it resolvable using /etc/hosts file of the current instance for all the servers in mysql cluster, and also change the whitelist IP range. Change the UUID to be the same as the first server

Restart MySQL server

sudo systemctl restart mysql

Login to mysql on master server (first server)

sudo mysql -u root

Run the following commands

Turn off bin log

SET SQL_LOG_BIN=0;

Create replication user

CREATE USER rpl_user@’%’ IDENTIFIED BY ‘123456789’;

Give appropriate permission

GRANT REPLICATION SLAVE ON *.* TO rpl_user@’%’;

Flush permissions

FLUSH PRIVILEGES;

Turn on bin log

SET SQL_LOG_BIN=1;

Enable user for replication

CHANGE MASTER TO MASTER_USER=’rpl_user’, MASTER_PASSWORD=’123456789′ FOR CHANNEL ‘group_replication_recovery’;

Start the replication

START GROUP_REPLICATION;

Disable read only mode

SET GLOBAL super_read_only=0;

Check if the node is online

SELECT * FROM performance_schema.replication_group_members;

Cluster third MySQL server to the group

Edit configuration

Open the file

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

Add the following content

disabled_storage_engines=”MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY”
server_id=3
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=binlog
binlog_format=ROW
plugin_load_add=’group_replication.so’
transaction_write_set_extraction=XXHASH64
group_replication_group_name=”40d2393e-a59b-11ea-8acd-0242ac110003″
group_replication_start_on_boot=off
group_replication_local_address= “service-db3:33061″
group_replication_group_seeds= “service-db1:33061,service-db2:33061,service-db3:33061″
group_replication_bootstrap_group=off
group_replication_ip_whitelist=”service-db1,service-db2,service-db3”

The server_id changes to 2 and 3 and group_replication_local_address changes to a resolvable url or IP address of the current instance for all the servers in mysql cluster, and also change the whitelist IP range. Change the UUID to be the same as the first server

Login to mysql on master server (first server)

sudo mysql -u root

Run the following commands

Turn off bin log

SET SQL_LOG_BIN=0;

Create replication user

CREATE USER rpl_user@’%’ IDENTIFIED BY ‘123456789’;

Give appropriate permission

GRANT REPLICATION SLAVE ON *.* TO rpl_user@’%’;

Flush permissions

FLUSH PRIVILEGES;

Turn on bin log

SET SQL_LOG_BIN=1;

Enable user for replication

CHANGE MASTER TO MASTER_USER=’rpl_user’, MASTER_PASSWORD=’123456789′ FOR CHANNEL ‘group_replication_recovery’;

Start the replication

START GROUP_REPLICATION;

Disable read only mode

SET GLOBAL super_read_only=0;

Check if the node is online

SELECT * FROM performance_schema.replication_group_members;

That’s it. Your MySQL servers are now clustered with redundancy.

Categories
Applications Linux Lsync NFS Rsync Ubuntu UFW

NFS server with Lsyncd in Ubuntu

Make the attached volumes ready

Create folders

sudo mkdir /mnt/upload_disk1sudo mkdir /mnt/upload_disk2sudo mkdir /mnt/backup_disk1sudo mkdir /mnt/backup_disk2

Create partition

sudo fdisk /dev/nvme0n1sudo fdisk /dev/nvme1n1sudo fdisk /dev/nvme2n1sudo fdisk /dev/nvme3n1

Make filesystem

sudo mkfs.xfs /dev/nvme0n1p1sudo mkfs.xfs /dev/nvme1n1p1sudo mkfs.xfs /dev/nvme2n1p1sudo mkfs.xfs /dev/nvme3n1p1

Add fstab entry

sudo vim /etc/fstab

Add the following content

/dev/nvme0n1p1 /mnt/upload_disk1 xfs defaults,nofail 0 0
/dev/nvme3n1p1 /mnt/upload_disk2 xfs defaults,nofail 0 0
/dev/nvme1n1p1 /mnt/backup_disk1 xfs defaults,nofail 0 0
/dev/nvme2n1p1 /mnt/backup_disk2 xfs defaults,nofail 0 0

Setup NFS

Update apt repo

sudo apt update

Install NFS server

sudo apt install nfs-kernel-server

Add configuration for NFS

sudo vim /etc/exports

Add the following content

/mnt/upload_disk1 *(rw,async,no_subtree_check,no_root_squash,no_all_squash)/mnt/backup_disk1 *(rw,async,no_subtree_check,no_root_squash,no_all_squash)

Change permission

sudo chmod 777 /mnt/*

Restart NFS server

sudo systemctl restart nfs-kernel-server.service

Setup backup using Lsyncd

Install Lsyncd

sudo apt install lsyncd

Make directory for configuration

sudo mkdir /etc/lsyncd

Create configuration file

sudo vim /etc/lsyncd/lsyncd.conf.lua

Add the following content

—— User configuration file for lsyncd.
—- Simple example for default rsync, but executing moves through on the target.
—- For more examples, see /usr/share/doc/lsyncd*/examples/–settings
{
    logfile = “/var/log/lsyncd/lsyncd.log”,
    statusFile = “/var/log/lsyncd/lsyncd.status”,statusInterval = 2
}
— Slave server configuration
sync {
       default.rsync,
       delete= false,
       source= “/mnt/upload_disk1”,
       target= “/mnt/upload_disk2”,
       rsync={
                archive = true,
                verbose  = true
             }
}
sync {
       default.rsync,
       delete= false,
       source= “/mnt/backup_disk1”,
       target= “/mnt/backup_disk2”,
       rsync={
                 archive = true,
                 verbose  = true
             }
}

Restart service

sudo systemctl restart lsyncd

Setup firewall rules

sudo ufw allow from 172.32.0.0/24 to any port nfs