Version control and Linux

Here’s an excerpt of what I have learnt from crio #learnbydoing challenge about version control and how to contribute to open source.

Linux commands

Linux is the best platform hands down, followed by mac (I’m not adding windows to the list, because it doesnt deserve it). If youre looking to develop stuff or work on a project Linux is the go to OS. In order to better know how to take advantage of the Linux system, you need atleast some understanding of file structures and a handful of basic commands.

know your location

To know where you are in your computer workspace use

pwd

This gives a directory structure which shows where you are.

in linux locations are separated by /

eg. /home/user/Documents

ls -a

this command lists all files within the current working folder.

change location

cd the_folder

this takes you to the the_folder directory

cd ..

this takes you to the previous folder in the structure

cd ~

takes you to the home directory

file/folder manipulation

In order to create a new folder

mkdir new_directory

creates a new folder named new_directory

cp old_location new_location

this command takes the file or folder from the first argument and places it in the second argument

mv old_location new_location

to remove a folder/file

rm -rf folder_name

rm stands for remove and the -r stands for recursive meaning everything within the folder will be deleted , -f means no prompts will be displayed before deletion

But be careful before using these commands because there is no undo button

file creation

There are a lot of ways to manipulate files and creating them

to create a new file use

touch newfile.name

there are a variety of command line text editors that can be used to change and edit files. The most popular ones are vim and nano

nano file.name

this opens file.name file and you can edit this file from the command line itself.

Grepping

Grep is a tool which helps in searching for key words and patterns in the files you specify. This is extremely useful if you constantly lookup words in large files.

grep yourword file1.name and file2.name

this command searches for yourword in file1 and file2. there are a lot of flags which are out of the scope of this blog but worth looking into

Git

Git is a version control system, which means multiple people can work on the same project and have multiple versions concurrently and integrate it to the main project.

This is extremely useful in large open source projects that are being contributed to by hundreads of people at any given time. How to contribute to open source is discussed later.

The best way is to use github and signup with it. signup here github

Now let’s start with the basics. First install git on your platform(preferably linux) with the command that suits your system

On linux

sudo apt-get install git

On windows

If youre on windows the best way if to get git desktop or github desktop and then using if from there. Another workaround is to get a linux subsystem running with instructions from this blog https://www.windowscentral.com/install-windows-subsystem-linux-windows-10

On mac

If you have xcode cli installed you can check if git is installed with

git --version

Or if you use homebrew just type

brew install git

With everything set up and ready, assign your identity on git with

to set your username

git config --global user.name "YOUR_USER_NAME"

to set your mail id

git config --global user.email "YOUR_MAIL_ID"

Starting your first project

The best way to start a project is going to github.com and creating a new project

now press on the download code button in green and copying the url.

In terminal open the location where you want your project to lay and type

git clone the_url_you_copied

This will create a new folder with the same name as your project name . move into this folder and youll find no files beacause you started a new project. Now add your project files to this folder.

Now you have added the files but git does not know which files to track and which not to. so in order to add the files to your tracking list

git add .

adds all files you have in your folder to the tracking list, to add just a file,

git add yourfile.name

The next step is to tell git that all files have been changed. You do this by

git commit -m "message"

This command commits (solidifies changes) all the changed files that have been tracked. -m is the message you give to your commit, which makes it easier for others working on the project know what the commit is about. (it makes it easier for you too). There is more than one way to do the same thing everywhere, you can research into this more.

Now we push the changes to our git space with

git push

now if you go to your github profile you will see your changes reflected in your repository (project folder).

Congrats, you have just created and committed your first change!

Checking out others projects

You can also clone and try out other people’s open projects. This is done the same way you do your git clone. In order to contribute a feature or a creative new idea to someone elses projects, you need to fork their repository. But do NOT push changes directly to their main branch(called master), but create a new branch and then push changes which if the original repo owner likes your changes will be pushed to the main repository.

And there you go, a brief intro to version control and the linux OS. Although we havent even scratched the surface of the glass, these should help you move out and get things done. Research into topics that interest you.

I learnt a lot from this micro experience at #crio #ibelieveindoing

Setting up your server with apache2 and wsgi

Having designed a portfolio site with django, I never imagined hosting a server would be so annoying and confusing. So here’s absolutely everything you need to know, do and should’nt when setting up your website with apache2 and python framework.

Assumptions

I assume that you have a django website ready. If you don’t, https://docs.djangoproject.com/en/3.0/intro/tutorial01/ is a good place to start. Ill also assume you have a virtual machine in aws and have connected to it through ssh, check https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html. The server we will be discussing is a ubuntu server, and not windows. Python3 will be used everywhere.

Installing the required

Apache2 is a well tested web server software that can serve your content to users. Some of you might be wondering why use apache when django provides a server on its own. But this server is a test server and cannot be used for deployment. Django is in the business of building frameworks not servers(their words not mine). So we need to use apache to get the most out of our django application and actually host it. A list of applications necessary are,

Before installing anything always make sure all package names are updated using

sudo apt-get update
  • Apache2
sudo apt-get install apache2
  • Apache2-dev
sudo apt-get install apache2-dev
  • pip3
sudo apt-get install python3-pip

pip is a necessary tool for handling python packages. The 3 in python is crucial because all ubuntu systems ship with both python2 and python3 and by default python refers to python2. This will save you a lot of headaches later on in the set up. (It is also recommended that you explicitly use python3 whenever its used from now on)

  • virtualenv
python3 -m pip install virtualenv

Here using virtualenv is advantageous and offers more functionality than using the venv package. It also creates a activate_this.py file which makes it a lot easier to activate the virtual environment. The python3 is necessary everywhere for all this to work smoothly as you never want version mismatch in python.

  • mod_wsgi
python3 -m pip install mod_wsgi

wsgi stands for web server gateway interface, which creates a link between your server hosting application(apache) and your web framework application(django) and allows your app to be served. Although there are a lot of methods to install mod_wsgi and a lot of different standalone wsgi packages, it is recommended that you use pip.

  • Your django application git repo cloned

Get your django application onto github and clone it in your remote machine. from here on, I will refer to this app as thissite with this django structure and its location is probably /home/ubuntu/

thissite/
    mysite/
        manage.py
        mysite/
            __init__.py
            settings.py
            urls.py
            asgi.py
            wsgi.py
        polls/
            __init__.py
            admin.py
            apps.py
            migrations/
                __init__.py
            models.py
            static/
                polls/
                   #staticfiles
            tests.py
            views.py

this should be familiar if you have created a django app or visited the link at the top. and explaining this is not within the scope of this post. we will be dealing with and editing the wsgi.py and the settings.py files only.

  • django

Move into your thissite folder. Here were going to create a virtual environment. This is essential if you want to host multiple sites with the same server. A virtual environment is an isolated space where installing packages wont interfere with other such environments. To create a new environment named myvenv type from within thissite folder

virtualenv myvenv

this creates a new folder named myvenv which we will use to activate and deactivate the environment. To activate the environment,

source mvenv/bin/activate

now install django into your environment with

python3 -m pip install django

Now with everything installed and available we are ready to get into setting up our apache server.

Configuring

The first thing to do is point apache2 to our installed wsgi module. This is done by finding the location where pip has installed the mod_wsgi package

mod_wsgi-express module-config

This outputs the location of the package in the first line.

LoadModule wsgi_module /usr/local/lib/python2.7/site-packages/mod_wsgi/server/mod_wsgi-py36.so
WSGIPythonHome /usr/local/lib

Copy the entire first line (with ctrl+shift+c). Move onto folder , /etc/apache2/mods-available which is where all apache2 modules available for use are present. now create a new file named wsgi.load using your favourite editor. (nano and vi are good options). Then paste the line you copied onto the file and save.

Now that the module is pointed to the right place, in order to enable it use the command

sudo a2enmod wsgi

where sudo is to execute as root user, a2enmod is short for apache2 enable module. next is to restart the apache2 with new configuration.

sudo systemctl restart apache2

If everything went as per plan mod_wsgi is loaded into apache and were good to go. One way to check if wsgi has been loaded is by typing

apache2ctl -M

This gives a list of all loaded modules and the last one will be wsgi. Next step is pointing apache to the django directory and the wsgi.py file inside. This is done by moving into directory /etc/apache2/sites-available and creating a new file named mysite.com.conf (mysite.com is your domain name, that will be used to access your site). Inside this file place the following lines, which ill explain.

<VirtualHost *:80>
        ServerName mysite.com
        ServerAlias www.mysite.com
        ServerAdmin yourmailis@zzz.com

        DocumentRoot /home/ubuntu/thissite

        <Directory /home/ubuntu/thissite>
        Order allow,deny
        Allow from all
        Require all granted
        </Directory>

        Alias /static /home/ubuntu/thissite/mysite/polls/static

        <Directory /home/ubuntu/thissite/mysite/polls>
        order allow,deny
        Allow from all
        Require all granted
        </Directory>

        WSGIDaemonProcess myapp python-path='/home/ubuntu/thissite/myvenv-packages:/home/ubuntu/thissite/mysite'
        WSGIProcessGroup myapp
        WSGIApplicationGroup %{GLOBAL}

        WSGIScriptAlias / /home/ubuntu/thissite/mysite/mysite/wsgi.py
        <Directory /home/ubuntu/thissite/mysite/mysite>
        Order allow,deny
        Allow from all
        Require all granted
        </Directory>

        ErrorLog /var/log/apache2/error.log
</VirtualHost>

This might look like a lot of info but is actually pretty basic when broken down

The first part is where basic server details such as admin mail id and the server name(domain name) are set. The document root directory is the place where apache will look for your website, but be careful not to put any sensitive files here as anyone from the web can access these files if inside the folder, for django you could even point it to an empty folder

The directory tag is how apache is given directions to the folder.

Alias /static is used to give apache the static files for your project so it can serve it directly instead of django.

The last part is where a daemon process is started and is set to use the python virtual environment. This is the recommended way of using modules inside the environment and if you want to host multiple sites.

Finally the WSGIScriptAlias points to the wsgi.py file in your django web application. After all this is set use the command

sudo a2ensite mysite.com
sudo systemctl restart

now everything that needs to be configured in your apache side is done.

The next step is to chang the wsgi.py file in your django application. So move to /home/ubuntu/thissite/mysite/mysite and open wsgi.py file with a text editor.

python_home = '/home/ubuntu/thissite/myvenv'

activate_this = python_home + '/bin/activate_this.py'
exec(open(activate_this).read(),dict(__file__=activate_this))

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'portfolio.settings')

application = get_wsgi_application()

The extra bit of code takes advantage of the activate_this file in your virtual environment and activates it from there instead of from apache itself. Which makes it easier to host. The final setting is to change the Settings.py file. add this line at the end of the file.

STATIC_ROOT = '/home/ubuntu/thissite/mysite/polls/static'

and also change ALLOWED_HOSTS to

ALLOWED_HOSTS = ['*']

to allow any ip addresss. Finally assign the domain name to your local ip address by going to /etc and adding the following line to your hosts file

172.17.0.1 mysite.com www.mysite.com

So whenever your server recieves a request for your domain name your local host is actually served.(If youre using a domain name from godaddy or google domains make sure you edit the A files to point to your remote machines external ip. for more details on this visit my previous blog post.)

Now if all went good you should be able to see your beautiful server when you type your domain name. If something unusual happens check your error log file at /var/log/apache2/error.log

Here are all the complete references https://modwsgi.readthedocs.io/en/develop/getting-started.html https://www.shellhacks.com/modwsgi-hello-world-example /https://pypi.org/project/mod-wsgi/

heres my portfolio I built with the same cconfiguration mentioned here, http://ramprakash.blog/

Getting started with cloud systems

Learning system design with #crio.do

I have refrained from understanding how cloud computing works, for fear of being overwhelmed by the AWS beginner guide. But I got a chance to understand it from a different source. Heres what I learnt from the system design miro experience.

Setting up

The first step in designing is to rent a virtual machine from Amazon Web Services. Create a new AWS account and head to the launch EC2 instance page. While setting up your virtual machine, youll be asked to create a key which will be used to access your machine with a secure shell from your computer. This key needs to be kept secure and this can be done by using

chmod 400 key.pem

command which allows only the admin to access the key file. next connect to the VM with the ssh command on linux or using putty on windows. After gaining access, set up a HTTP server on your machine using Nginx, which allows you to configure and set up a fresh website.

In order to access your website a static IP (Ip that does not change every time you restart your machine) which can be done in the AWS from the elastic IP panel. This enables you to access your server from your web browser by typing the static ip address.

Diving in

Now that you have your server up and ready, we need a domain name to go with your server ( would you rather type google.com or 8.8.4.4 ?). After visiting a domain hosting site like godaddy.com ( and painstakingly finding a domain name that suits you) The static ip address of your server can now be assigned to your domain name. But everyone cannot access your website because AWS has inbuilt firewall that blocks all ports. So create a new access group and allow all http traffic through it from any ip. Now your website can be accessed with your domain name.

Actual system design

Now that we have our website ready to go, there needs to be a way to split up heavy traffic in your servers. To handle this set up a new VM the same way as before and configure it a static ip. Now in your domain name provider settings add this new ip to your domain name. This means when a person accesses your domain name, either of the two ip addresses gets served based on the current traffic (if one server is down, the other will be served).

Final content delivery

With everything set up, if your website wants to deliver humongous files or large content, a content delivery netword will reduce the time taken to serve it to a user.So set up an account with some free cdn providers. The basic idea of a cdn is creating cached version of your content in multiple parts of the world so that the content can be accessed by people from various parts without much delay.

Thoughts

This micro experience was a fun way to explore new territories and the #IBelieveInDoing workshop was a deep and enlightening experience. Glad to have been a part of this wornderful endeavour.

Emission, volumetrics and volume shaders

These are two closely related shaders that go well with each other. Lets start with the emission. It makes an object glow with a particular colour. This is useful when the lighting used in a scene is shaped a particular way(like those halo lights streamers use). This gives the scene the specific look imagined.

If you’ve seen a rendered image and something made it look ethereal or punchy, its probably the emission shader.

The way this works with volume shaders is crazy. Volume shaders differ from normal shaders. Normal surface shaders give a look only to the surface of the object. It does not affect the size or the density of the object. Here’s where volumes come in. Volume shaders give the entire volume a particular look. It can be used to determine how much light enters the object(imagine light shone on ice).

When the density of the volune is given a small number like 0.05 to .5, this gives the entire object a slight foggy look. The anisotropy value is set to about 0.3. Anisotropy is basically getting a different look from different angles(this is such a deep topic that deserves its own post).

When another object with emission shader is placed inside the volume object, voila you get cool rays from the emission object that give the scene depth instead of just being normal lighting. This is called volumetrics where you can actually view the light direction. There is a world volume shader which shades the entire world with the particular volume.

So, it’s no surprise that most of my renders have crazy colored emissions and volumetrics.

Freaky Wireframe

Let’s talk about the wireframe modifier. Looks like just another boring modifier but honestly one of the more powerful modifiers available.

It takes a solid object and converts it into another solid mesh with the original object’s edges. The more you difficult the merrier.

The offset value determines if the mesh appears inside or outside the original object, essentially allowing you to create cool effects with a duplicated object along with the original. More cool effects if used with the boolean modifier

The vertex group provides an area in which the modifier is to be applied

Thickness factor determines the sharp or round edges for the new mesh work that is created.

Replace original replaces the original mesh with the created mesh essentially creating a new object.

That’s all there is to know about this particular modifier but not all of it’s applications. One crazy use i found was creatting veins in a leaf. The top modifier on my favourites list only after the wave modifier.

Bellman, markov and the q

Surfing through some hackerrank profiles, I found a guy, who made an AI chess addon that can view any chess board in a browsre page and provide a continuation for it. Which got me really interested in AI and spending money on udemy.

The concept that was the base of all the AI magic was the bellman’s equation.

It describes that the value of any state of an environment as the maximum sum of reward for performing an action and the value of the next state.

The bellman’s equation goes like so

V(s) = max(R(s,a) + ∆V(s’))

But this equation is for static environments with no probability.

For environments with a probability factor, markov equation gives an equation with consideration of randomness

V(s) = max(R(s,a) + ∆*sum(p(s,a,s’)V(s’)))

But even better is the concept of q learning. The base concept of q learning is that, instead of giving values to a state and action, we definitely the quality of an action. This has the effect of separating each state based on the actions it can take and prescribing the best action to take also considering random probability.

Q(s,a) = R(s.a) + ∆*sum (p(s,a,s’)*max (Q(s’,a’)))

This recursively gives the action based on previous action quality, thereby providing more accurate Intel and paths to traverse the environment.

The temporal difference is yhe difference between previous q value and newly calculated q value based on new information. This temporal difference is dynamic and chooses best action based on previous and current data

TD(a,s) = R(s, a) + ∆max (Q(s’, a’) – Qt-1(s, a))

And

Qt(s,a) = Qt-1(s,a) + ∆TDt(a,s)

=> Qt(s,a) = Qt-1(s,a) + @(R(s, a) + ∆max (Q(s’, a’) – Qt-1(s, a)))

Which may look complicated but simple enough and means that current Q value depends on previous Q values and the current reward thereby changing any value dynamically based on rewards acquired. If the new value is low the old value is kept as such. Rewards can be given at any state of the environment, gaining accurate control and precision over the agent

R – reward

V – value

Q – quality

P – probability of an action (or randomness in environment)

TD – temporal difference

You can always teach a new dog new tricks, if it’s willing to learn.

Domains and names

The domain name system, DNS for short is one of those creative ideas that make human life easier on the internet. The system is for converting human understandable domain names into ip addresses for easy relembrance and searching of a particular ip address.

There are four important parts for a smooth domain retrieval. They are the recursive DNS retriever, the Top Level Domain nameserver, root nameserver and authoritative nameserver.

The recuesive retriever is the first one to recieve the user input ‘theserver.com’ this then queries the DNS root nameserver.

This then sends the query to a TLD nameserver which has the .com level domain.

The TLD replies with the domain nameserver consisiting of theserver.com domain name which then sends the ip address of the domain.

Finally the DNS resolver responds to the user browser by providing the ip address found, and renders the webpage that is to be shown to the user.

The crazy cool thing is how a computer figuratively understands human language does what needs to be done by itself.

Vacuum and tubes

Disassembling an old tape recorder, I found there were tiny bulb structures that didn’t seem to light up (or have tungsten for that matter). Upon research I was surprised to find that there was a time in history where silicon was just sand.

This was the period of vaccum tubes. This is a creative engineering solution for computing problems.

The tube consists of two electrodes(as usual) the cathode and anode. The cathode is heated which produces electrons due to a phenomenon called thermionic emission. The concept is that metals emit electrons when heated. These electrons reach the anode due to the positive potential at the anode. This creates a currentflow from anode to cathode(electricity flows opposite to electrons).

Thereby it essentially creates a diode (di-electrode??). Diodes conduct in one direction only. This diode was used mainly for rectification of ac to dc.

For transistors, a thurd electrode was installed in between the cathode and anode. This was a metal mesh which was given an external negative supply voltage. This negative voltage repels the electron flow. By controlling the voltage level we can essentially control the circuit current creating a triode.

Triodes are mainly used as amplifiers because they are in essence an electric switch. This triode was also used to construct logic gates.

There was even a variant with four and five electrodes, the two additional eoectrodes called screen grid and suppression grid for further control of electron flow.

The entirety of the tube is kept in vaccum to provide clear paths for the electrons.

Now that, is something to marvel about.

A problem with binary

Confusion arose when the lights turned on when they were supposed to be off and lit the room blue.

The LED was connected to pin 18 of the gpio and given an on pulse. But nothing interesting happened. When the off signal was given, the LED started off.

Upon investigating the problem in the strange waters of the internet, there were a lot of confusing answers and I realized the fact that a basic understanding of how LEDs and gpio works was essential.

GPIO input/output pin works by sending a 1 for on pulse and a 0 for off pulse. 1 represents sourcing and 0 represents sinking current. So 1 doesn’t necessarily mean turning the connected device on. It just means a current is sourced.

If the anode of the LED is connected to the I/O port the port can provide only a 12mA current source. This means th LED doesn’t have enough current.

But if the cathode was connected to I/O instead, and the anode to the 3V supply, (more than 20 mA) full brightness can be reached but the configuration is inverted and LED turns on, when the 0 pulse is given and off when the 1 pulse is given.

The best way to adress this boggling issue is to change the configuration in the code from 1 for on, to 0 for on. Although this sounds like a workaraound, truth is, no one cares how it works and just wants to see that light glow

Internet protocols

Therw are two protocols for internet traffic.

TCP(transmission control protocol) is the safe and secure protocol. It engages a three way handshake with the recipient. The first phase is the SYN(chronous) where it establishes and verifies connection with the neighbour. The second phase is the SYN ack(synchronous acknowledgement) where the recipient computer ensures connection. The third phase is acknowledgement when the sender computer acknowledges the SYN ack and starts sending data. This method is slow and takes a lot of bandwidth but is extremely secure. If there is a loss of data packet, TCP asks the sender to resend that particular packet

But, UDP is all about dat speed. It doesn’t even verify SYN, but keeps on sending packets without any concern for the recieving computer. The advantage is it’s high speed and bandwidth.

Personally, I prefer UDP, because loss or frame has a very low probability of occouring and ofc everyone prefers fast over secure