Categories
Applications AWS EC2 Firewall Linux NethServer OpenVPN Roundcube Security/Vulnerability

NethServer on AWS

If you ever wanted to setup a VPN server, mail server, mail box, web server, or nextcloud (an open source google drive alternative), then you can go through this blog and set it up in minutes using NethServer.

NethServer new management web UI at 9090 port

Since AWS does not provide a NethServer AMI by default via MarketPlace or Community AMI, I had to make a VM image in my local server, modify it, import it to AWS and then modify it again and make it working for AWS.

I have made that image public and it is available in the Mumbai region. If you need it in any other region, contact me or you can make an AMI after launching the server in the Mumbai region and transfer it to whichever region you like.

So to setup a NethServer in AWS, below are the steps you have to follow

Setting up EC2

The instance

Go to the AWS EC2 console and click on Launch Instance

There go to Community AMIs and search for NethServer or you can search the below AMI ID. Do note that this AMI ID might change if I update the AMI

ami-0c0dc2c9b42edfa60

Security group rules

The security group should allow the following as per requirement for accessing the server

For the latest management web UI, open the port 9090 inbound

For old management web UI, open port 980 inbound

For SSH Access, open port 22 inbound

You can keep outbound access as full open for all traffic.

User Access

The NethServer AMI by default comes with a user. The details are as follows

Username: admin
Password: Nethserver@123

Domain/Other details

By default, this AMI comes with the FQDN ami.vigneshn.in

This can be changed and there are also my contact details in the Company name space so you can change it according to your requirements.

Also note that you have to change the FQDN before you install LDAP in account providers (Users and Groups).

You can also use Active directory (external) for user and group management.

Also you need to setup LDAP to change the password of the default admin user

Known caveats

Currently there is an issue with Nethserver where it requrires a green interface at any cost, without which it will throw and error at ipconf step.

To fix this, create a network interface in AWS and attach it to the instance.

Make this a green network and if your ip is 172.20.20.20 for the network interface, then give that as static in for green and the subnet mask as 255.255.255.0 and the gateway as 172.20.20.1

Documentation

For more details on how to set up the individual services in NethServer, you can visit the NethServer documentation here.

Categories
AWS EC2 Gaming GPU Instance NVIDIA Parsec Windows

PC gaming without a rig – using Parsec on Windows!

If you are like me, building a gaming rig would be one of your dreams, but to build a rig that can play high end games on ultra graphics would cost you an arm and a leg.

If you are stuck in this situation, then read on to find out how to start gaming without building a rig, and no, this is not about gaming services like stadia or shadow which are not available in India yet.

Requirements

There are a few requirements that I’ll mention below that you’ll need to get this working properly.

  • AWS Account – Basic knowledge on how to launch servers
  • A good internet connection – 30-40mbps – FTTH (for low latency)
  • A job/business to pay the bill – AWS isn’t free, but it’s cheap
  • A game – (Always buy your games when they are high end because otherwise the developers would feel that they can make more money from s**t games like candy crush and they’ll stop making good games. We really don’t want that.)
  • Local machine running any of the following
    • Windows 7+
    • Google chrome
    • MacOS 10.11+
    • Android – With google play
    • Raspberry pi 3
    • Linux – Ubuntu 18.04+

Let’s get started

So to get started, first we need a Parsec logo account and then we will move on to setting up the game server in AWS.

Setting up Parsec

What is parsec?

So Parsec is like VNC/RDP on steroids. They give very low latency interactive streaming from a remote PC to your local machine.

They say they use a protocol called BUD(Better User Datagrams) which is like UDP on cocaine and they developed in house specifically for gaming but I think they are using magic and this new protocol talk is just to cover up the magic.

Seriously!

I hosted my server in Mumbai region in AWS and I live in thrissur, Kerala and the LOS distance between my house and Mumbai is 1100 km, so it should be twice that in network fibre length and with all those switched and routers in between, I expected it not to work but it worked! The latency I got is only 30-40 ms which is basically like gaming on your local machine. You won’t feel it. 30 ms is approximately a lag of 1 frame at 30fps which is nothing.

You can read more here

Creating an account

To setup a parsec account first sign up here. It is a very simple process

First give a username

The they’ll ask for an E-mail ID and password

Always use a strong password and get the achievement unlock

Now you’ll get a confirmation E-mail, so click the confirmation link and your account is ready.

The client

Now that your account is ready, it is time to download the client (or you can use their web client) and login.

You can download the appropriate client for your platform from here

Setting up server in AWS

You always have an option to launch an EC2 instance in the classic method i.e without VPC but I will not recommend that so for this setup, we’ll start with VPC and its sub services and then move on to create the server.

Setting up a VPC

I have covered this in my blog on serverless wordpress but I will cover the basics again here.

First login to your AWS account and go to the VPC console

Create a VPC here by giving the proper CIDR block and we don’t need IPv6 for this so you don’t have to give that but give a Name tag so that we can understand what it’s for.

Next create an internet gate way and attach it to the VPC

Then create a subnet for the EC2 instance.

Once that is done create a Public route table (we don’t need a private route table for this one) and add the internet gateway for 0.0.0.0/0 route and also associate the created subnet

Setting up EC2

To setup the EC2 instance, first create the security group, then subscribe the required AMI from the market-place and then we have to launch the instance

Security group

Go to the security group tab in the EC2 console and enter the following rules

Inbound
All traffic | All | All | Anywhere | “0.0.0.0/0” / “::/0”
This is not at all secure but its convenient.
You can check here and here to get the exact port requirements.

Outbound
All traffic | All | All | Anywhere | “0.0.0.0/0” / “::/0”

AWS Marketplace

Since parsec does not support hosting on Linux yet, we have to use Windows. So you can subscribe the NVIDIA Gaming PC – Windows Server 2019 from the AWS Marketplace.

Login to your AWS account, go to the AWS marketplace console and subscribe the AMI.

Setting up the Server

Once the AMI is subscribed, you can launch your instance with the following configuration

AMI – NVIDIA Gaming PC – Windows Server 2019

Choose instance type is g4dn.xlarge (which is the smallest that you can choose)

VPC – Newly created VPC
Subnet – Newly created subnet
Enable Public IPv4

Shout out to my buddies at LaresAI – I’m creating this in their dev env VPC

Add a 50G-60G of additional storage – You will see a 125GB of storage by default but that is ephemeral and you’ll lose the data if the instance is stopped and started. If you add additional storage, you’ll have to format it after boot up. Otherwise you can also increase the C Drive storage from 30GB to any value that you like (Provided you are ok paying for it).

Now give a name tag which you can understand

Now specify the security group that you already created

That’s it. Now you can review and launch that instance.

You have to wait for some time for the instance to be launched and then you can get the password for the Administrator account for the Windows server.

Logging in

To login to the Windows server, go to your EC2 console, select the server and click on connect.
You will see an option to upload or copy/paste the key they you had selected when you launched the instance.

This key is required to get the automatically generated Administrator password.

If it shows some error, you have to wait for some more time for the instance to complete the setup.

Once you get the password, you can use Remmina or any RDP client to connect to the server with it’s public IP.

Once connected, install Parsec and login the same way you logged in in your client and ensure that the Hosting option is enabled

This will show your PC in parsec and you can see the same in your client

Connecting with Parsec and setting up sound

Not that the parsec is connected, you have to do a few things to get it up and running for gaming.

Disconnect from RDP and connect again using Parsec from your client. It should work. If it does not, make sure to check the trouble-shooting steps as mentioned in the Parsec website or join Discord.

Once your Parsec is connected, you’ll notice that there is no sound. This is because AWS does not attach a sound hardware since it is a server. To fix this, you can install a virtual sound hardware from here.

Click on the Install batch file to install the virtual sound drive. You can enabled spacial sound for better immersive experience while playing games.

Now you are ready to play any game

Gaming

Now you can install Steam or any gaming platform.

I’m using steam. I installed Shadow of the tomb raider

Once installed you can start playing it like it is your local PC but with graphics on ultra.

You can see the screenshots from my gameplay taken from my laptop below.

Categories
Applications AWS CentOS Linux Virtual Machine Manager Virtualization

Setting up a CentOS 8 server as a virtualization host

If you are trying to setup virtual machines in a CentOS 8 server, the following are the steps.

I set this up in AWS with an m5d.metal instance (only metal instances allow direct hardware access in AWS. AWS does not support nested virtualization) but it is the same for any CentOS 8 server.

Once you get the server up and running, check if your hardware supports virtualization. If it is a new machine with a recent processor, it should support it but just check it just in case.

grep -E '(vmx|svm)' /proc/cpuinfo
lsmod | grep -i kvm

Now we install the X-Org server so that you can access the virt-manager GUI in the server on your local machine.

sudo yum install xorg-x11-xauth xorg-x11-fonts-* xorg-x11-utils

Now install all the required packages required for virtualization

sudo yum groupinstall "Virtualization Host"

Once the installation is complete, check the status of libvirtd service which is required to run the virtual machine.

sudo systemctl status libvirtd

If it is not running, then run the following command to start the service

sudo systemctl start libvirtd

Remember to enable the service so that the service starts up automatically after reboot

sudo systemctl enable libvirtd

In the current situation, only root can connect to the libvirtd daemon to create VMs but since we are planning to use the GUI and not the virt-install command line tool, it will be better to add the user to the libvirt group. This is because root doesn’t play will with X-Org server.

sudo usermod -aG libvirt centos

Now you have to log off and log in.

You can run the following command to kill all the process by the user (here it is centos) which will basically log you out.

sudo pkill -U centos

You can log back in with the -X option so that when you run the virt-manager, you can see it in your local machine.

ssh -i key.pem -X centos@IP

Once you have logged in, check if you are there in the libvirt group by running the following command.

groups

Confirm that the installation of packages was completed successfull by running the following command.

sudo virsh version

Once all that is done, you can run the following command to start the GUI Virtual machine manager.

virt-manager

That’s it. You may now setup virtual machines in the server using the same steps you have been following till now with Virtual machine manager.

Categories
Applications Cluster Linux MySQL

MySQL-Cluster group replication

Follow this to setup a MySQL database cluster with group replication. Group replication is where the database across the cluster is constantly synced, so you can write or read from any database and it will be replicated across the other ones.

This increases the redundancy and read throughput.

Assumptions

Following are the assumptions that are made.

Hostname of service DB server 1: service-db1
Hostname of service DB server 2: service-db2
Hostname of service DB server 3: service-db3

IP of service DB server 1: 172.32.0.10
IP of service DB server 2: 172.32.0.20
IP of service DB server 3: 172.32.0.30

MySQL port: 3306
MySQL cluster port: 33061

Install MySQL on all three servers

Add the MySQL community repo

Download the deb file

curl -OL https://repo.mysql.com//mysql-apt-config_0.8.15-1_all.deb

Install using dpkg

sudo dpkg -i mysql-apt-config_0.8.15-1_all.deb

Select mysql-5.7

Update the apt repo

sudo apt update

Install MySQL server

sudo apt install mysql-server

Start MySQL service

sudo systemctl start mysql && sudo systemctl enable mysql

Do secure installation

sudo mysql_secure_installation

Open configuration

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

Edit configuration

OptionOld valueNew value
bind-address127.0.0.10.0.0.0

Add /etc/hosts entries to make hostnames resolvable

Open hosts file

sudo vim /etc/hosts

Add the following configuration after changing the IPs appropriately

172.32.0.10 service-db1
172.32.0.20 service-db2
172.32.0.30 service-db3

Add firewall configuration

sudo ufw allow from 172.32.0.54 to any port 3306 proto tcp
sudo ufw allow from 172.32.0.175 to any port 3306 proto tcp
sudo ufw allow from 172.32.0.166 to any port 3306 proto tcp

sudo ufw allow out to 172.32.0.54 port 3306 proto tcp
sudo ufw allow out to 172.32.0.175 port 3306 proto tcp
sudo ufw allow out to 172.32.0.166 port 3306 proto tcp

sudo ufw allow out to 172.32.0.54 port 33061 proto tcp
sudo ufw allow out to 172.32.0.175 port 33061 proto tcp
sudo ufw allow out to 172.32.0.166 port 33061 proto tcp

sudo ufw allow from 172.32.0.54 to any port 33061 proto tcp
sudo ufw allow from 172.32.0.175 to any port 33061 proto tcp
sudo ufw allow from 172.32.0.166 to any port 33061 proto tcp

Prepare first MySQL server for group replication

Edit configuration

Open the file

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

Add the following content

disabled_storage_engines=”MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY”
server_id=1
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=binlog
binlog_format=ROW
plugin_load_add=’group_replication.so’
transaction_write_set_extraction=XXHASH64
group_replication_group_name=”40d2393e-a59b-11ea-8acd-0242ac110003″
group_replication_start_on_boot=off
group_replication_local_address= “service-db1:33061″
group_replication_group_seeds= “service-db1:33061,service-db2:33061,service-db3:33061″
group_replication_bootstrap_group=off
group_replication_ip_whitelist=”service-db1,service-db2,service-db3”

The server_id changes to 2 and 3 and group_replication_local_address changes to the hostname and make it resolvable using /etc/hosts file of the current instance for all the servers in mysql cluster, and also change the whitelist IP range.

Login to mysql

sudo mysql -u root

Get UUID

SELECT UUID();

Replace UUID in the configuration file

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

The UUID to be changed is at group_replication_group_name section

Restart MySQL server

sudo systemctl restart mysql

Login to mysql

sudo mysql -u root

Run the following commands

Turn off bin log

SET SQL_LOG_BIN=0;

Create replication user

CREATE USER rpl_user@’%’ IDENTIFIED BY ‘123456789’;

Give appropriate permission

GRANT REPLICATION SLAVE ON *.* TO rpl_user@’%’;

Flush permissions

FLUSH PRIVILEGES;

Turn on bin log

SET SQL_LOG_BIN=1;

Enable user for replication

CHANGE MASTER TO MASTER_USER=’rpl_user’, MASTER_PASSWORD=’123456789′ FOR CHANNEL ‘group_replication_recovery’;

Bootstrap the replication group

SET GLOBAL group_replication_bootstrap_group=ON;

Start the replication

START GROUP_REPLICATION;

Stop the bootstrap

SET GLOBAL group_replication_bootstrap_group=OFF;

Check if the node is online

SELECT * FROM performance_schema.replication_group_members;

Change the mysqld.cnf file group_replication_start_on_boot to on

Cluster second MySQL server to the group

Edit configuration

Open the file

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

Add the following content

disabled_storage_engines=”MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY”
server_id=2
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=binlog
binlog_format=ROW
plugin_load_add=’group_replication.so’
transaction_write_set_extraction=XXHASH64
group_replication_group_name=”40d2393e-a59b-11ea-8acd-0242ac110003″
group_replication_start_on_boot=off
group_replication_local_address= “service-db2:33061″
group_replication_group_seeds= “service-db1:33061,service-db2:33061,service-db3:33061″
group_replication_bootstrap_group=off
group_replication_ip_whitelist=”service-db1,service-db2,service-db3”

The server_id changes to 2 and 3 and group_replication_local_address changes to the hostname and make it resolvable using /etc/hosts file of the current instance for all the servers in mysql cluster, and also change the whitelist IP range. Change the UUID to be the same as the first server

Restart MySQL server

sudo systemctl restart mysql

Login to mysql on master server (first server)

sudo mysql -u root

Run the following commands

Turn off bin log

SET SQL_LOG_BIN=0;

Create replication user

CREATE USER rpl_user@’%’ IDENTIFIED BY ‘123456789’;

Give appropriate permission

GRANT REPLICATION SLAVE ON *.* TO rpl_user@’%’;

Flush permissions

FLUSH PRIVILEGES;

Turn on bin log

SET SQL_LOG_BIN=1;

Enable user for replication

CHANGE MASTER TO MASTER_USER=’rpl_user’, MASTER_PASSWORD=’123456789′ FOR CHANNEL ‘group_replication_recovery’;

Start the replication

START GROUP_REPLICATION;

Disable read only mode

SET GLOBAL super_read_only=0;

Check if the node is online

SELECT * FROM performance_schema.replication_group_members;

Cluster third MySQL server to the group

Edit configuration

Open the file

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

Add the following content

disabled_storage_engines=”MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY”
server_id=3
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=binlog
binlog_format=ROW
plugin_load_add=’group_replication.so’
transaction_write_set_extraction=XXHASH64
group_replication_group_name=”40d2393e-a59b-11ea-8acd-0242ac110003″
group_replication_start_on_boot=off
group_replication_local_address= “service-db3:33061″
group_replication_group_seeds= “service-db1:33061,service-db2:33061,service-db3:33061″
group_replication_bootstrap_group=off
group_replication_ip_whitelist=”service-db1,service-db2,service-db3”

The server_id changes to 2 and 3 and group_replication_local_address changes to a resolvable url or IP address of the current instance for all the servers in mysql cluster, and also change the whitelist IP range. Change the UUID to be the same as the first server

Login to mysql on master server (first server)

sudo mysql -u root

Run the following commands

Turn off bin log

SET SQL_LOG_BIN=0;

Create replication user

CREATE USER rpl_user@’%’ IDENTIFIED BY ‘123456789’;

Give appropriate permission

GRANT REPLICATION SLAVE ON *.* TO rpl_user@’%’;

Flush permissions

FLUSH PRIVILEGES;

Turn on bin log

SET SQL_LOG_BIN=1;

Enable user for replication

CHANGE MASTER TO MASTER_USER=’rpl_user’, MASTER_PASSWORD=’123456789′ FOR CHANNEL ‘group_replication_recovery’;

Start the replication

START GROUP_REPLICATION;

Disable read only mode

SET GLOBAL super_read_only=0;

Check if the node is online

SELECT * FROM performance_schema.replication_group_members;

That’s it. Your MySQL servers are now clustered with redundancy.

Categories
Applications Linux Lsync NFS Rsync Ubuntu UFW

NFS server with Lsyncd in Ubuntu

Make the attached volumes ready

Create folders

sudo mkdir /mnt/upload_disk1sudo mkdir /mnt/upload_disk2sudo mkdir /mnt/backup_disk1sudo mkdir /mnt/backup_disk2

Create partition

sudo fdisk /dev/nvme0n1sudo fdisk /dev/nvme1n1sudo fdisk /dev/nvme2n1sudo fdisk /dev/nvme3n1

Make filesystem

sudo mkfs.xfs /dev/nvme0n1p1sudo mkfs.xfs /dev/nvme1n1p1sudo mkfs.xfs /dev/nvme2n1p1sudo mkfs.xfs /dev/nvme3n1p1

Add fstab entry

sudo vim /etc/fstab

Add the following content

/dev/nvme0n1p1 /mnt/upload_disk1 xfs defaults,nofail 0 0
/dev/nvme3n1p1 /mnt/upload_disk2 xfs defaults,nofail 0 0
/dev/nvme1n1p1 /mnt/backup_disk1 xfs defaults,nofail 0 0
/dev/nvme2n1p1 /mnt/backup_disk2 xfs defaults,nofail 0 0

Setup NFS

Update apt repo

sudo apt update

Install NFS server

sudo apt install nfs-kernel-server

Add configuration for NFS

sudo vim /etc/exports

Add the following content

/mnt/upload_disk1 *(rw,async,no_subtree_check,no_root_squash,no_all_squash)/mnt/backup_disk1 *(rw,async,no_subtree_check,no_root_squash,no_all_squash)

Change permission

sudo chmod 777 /mnt/*

Restart NFS server

sudo systemctl restart nfs-kernel-server.service

Setup backup using Lsyncd

Install Lsyncd

sudo apt install lsyncd

Make directory for configuration

sudo mkdir /etc/lsyncd

Create configuration file

sudo vim /etc/lsyncd/lsyncd.conf.lua

Add the following content

—— User configuration file for lsyncd.
—- Simple example for default rsync, but executing moves through on the target.
—- For more examples, see /usr/share/doc/lsyncd*/examples/–settings
{
    logfile = “/var/log/lsyncd/lsyncd.log”,
    statusFile = “/var/log/lsyncd/lsyncd.status”,statusInterval = 2
}
— Slave server configuration
sync {
       default.rsync,
       delete= false,
       source= “/mnt/upload_disk1”,
       target= “/mnt/upload_disk2”,
       rsync={
                archive = true,
                verbose  = true
             }
}
sync {
       default.rsync,
       delete= false,
       source= “/mnt/backup_disk1”,
       target= “/mnt/backup_disk2”,
       rsync={
                 archive = true,
                 verbose  = true
             }
}

Restart service

sudo systemctl restart lsyncd

Setup firewall rules

sudo ufw allow from 172.32.0.0/24 to any port nfs