Running a Python or Django Application with Supervisor

Making use of Supervisor

I have lurking around to find a example how can I get my python application work with Supervisor and not able to find some good examples so decided to write one. it will be brief.

Install supervisor

Go with

pip install superviosr

and it’s done. If not please check the official documentation.

Configure the Supervisor

Supervisor have 3 utility.

1. supervisord
2. supervisorctl
3. Web server

supervisord (supervisor daemon) run in background and process all your configuration.
supervisorctl a CLI that can be used to to access some information regarding running process, configuration change, re-start the supervisor daemon etc.
Web server can check the stack/log for the running services.

After installing the supervisor run

echo_supervisord_conf

It will show a sample of supervisor config. Read through it to get some insight.

Let’s create a configuration file that can be used to start the supervisor. (you can copy-past echo_supervisord_conf content and play around that).

In the Installation folder (/etc/supervisor in Ubuntu), do following:–>

create a supervisor.conf file (vim supervisor.conf) with following content

let’s get it one by one :–>

1. unix_http_server

   Start of configuration

   ; supervisor config file
   [unix_http_server]
   file=/var/run/supervisor.sock
   chmod=0700
   

2. supervisord

   This section content the functioning configuration of supervisor daemon. Here we have configured below
   a. logfile --> path of file to record log of supervisor daemon
   b. pidfile --> file in which the process id will be logged.
   c. childlogdir --> path of file to record the child process that are running through supervisor daemon.

   [supervisord]
   logfile=/var/log/supervisor/supervisord.log
   pidfile=/var/run/supervisord.pid 
   childlogdir=/var/log/supervisor 
   

3. inet_http_server and rpcinterface:supervisor:

   This section is used to define the web interface of the supervisor. Here we are defining the port and rpinterface

    [inet_http_server]
    port = 127.0.0.1:9001
    
    [rpcinterface:supervisor]
    supervisor.rpcinterface_factory=supervisor.rpcinterface:make_main_rpcinterface
   

4. supervisorctl

   Define configuration related to supervisor. Let’s define the server URL for now.

   [supervisorctl]
   serverurl = http://127.0.0.1:9001
   

5. Include section

   Include section to include the addition programs/process related configuration that we are going to run using the supervisor.

    [include]
    files = /etc/supervisor/conf.d/*.ini
   

Running a Python Script using Supervisor

Create a simple python script (sampleapp.py)

import logging
from random import randint
from time import sleep

def main():
    while True:
        i = randint(1,100)
        # If get a digit in multiple of 10 then crash
        if i in [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]:
            logging.error('Generated {}. Application Crashing'.format(i))
            raise Exception('Application Crashing')
        else:
            print('Generated {}. Sleeping for {}'.format(i, str(i//10)))
        sleep(i//10)

if __name__ == "__main__":
    print('Starting the sample app')
    main()

Now define a sampleapp.ini file

• define a program using [program:program_name]

• directory - define the directory where your script is ( Program will change to mentioned directory while running the program.

• command- What to run

  [program:sample_app]
  directory=/path_to_script_containing_folder/
  command=python3 sample_app.py

In Include section we have mentioned that /etc/supervisor/conf.d/*ini will be considered. So either put this sampleapp.ini file in /etc/supervisor/conf.d or create a symlink.

Now run

supervisord

It will start and run in background.

Now Run

supervisorctl

If everything is alright then you will see a response similar to this and we will enter inside the CLI utility of supervisor.

sajjan# supervisorctl
sample_app   RUNNING   pid 25366, uptime 0:01:34
supervisor> 
supervisor> help
default commands (type help <topic>):
=====================================
add    exit      open  reload  restart   start   tail   
avail  fg        pid   remove  shutdown  status  update 
clear  maintail  quit  reread  signal    stop    version

Play around it. Now to check the web interface of the supervisor, visit http://127.0.0.1:9001

Running a Django Application using Supervisor

Similarly, we create a django.ini file for our Django application.

[program:application_name]
directory=/path_to_your_project/
command=/path_to_your_virtual_enviroment/python3 manage.py runserver

Restart the daemon and it’s done. Play around it’s different parameter in configuration file.

F-1 Score, Precision and Recall

Hi,

Initially, when I started to learn machine learning, I was not able to grasp the terms F-1 Score, precision and recall along with False Positive and False Negative( ya these 2 are more confusing then True Positive and True Negative).

Reading post per post and still not able to get anything in my mind. So One day I sat and just cleared it out. Ok, Fine Here are the Details. Let's start with abbreviation.
Let's Create the basic table which we see in every post.

		Predicted
		Yes	No
Actual	Yes	TP	FN
	No	FP	TN

TP -  True  Positive      ----> Predicted is Positive and that is True
TN - True  Negative   ----> Predicted is Negative and that is True
FP - False Positive    ----> Predicted is Positive and that is False
FN - False Negative  ----> Predicted is Negative and that is False

So In TP/TN/FP/FN ending Positive, Negative tell us about the prediction. and the Initial True, False tell us "Is that Correct or Not?"

So False Positive means that the prediction is "Positive" but and False Indicate it is Wrong. For example an data corresponds to  class 0 but our classifier predicted the class 1.

Now same can go for False negative. I am leaving that up to you.

Now Here comes the Precision and Recall.

Precision

precison = TP/(TP+FP)

                = (Correct Yes Predicted by Model)/ (Total Yes predicted by Model [TP+FP])

This tells us "How much our model predicted correctly out of Total True predicted by Model(TP+FP)"

Recall

recall = TP/(TP+FN)
         
         = (Correct Yes Predicted by Model)/ (Total Yes in Actual Data[TP+FN])

This tells us "How much our model predict correctly out of total True in Data"


Recall address the question: "Given the positive example, will the classifier detect it".
While Precision address the question: "Given a positive prediction from classifier, how likely is it to be correct".

The image is just for my personal purpose.

Python Generator and Iterator

Iterator

Iterator is an object which allows to traverse through all the elements of a collection.

In Python, an iterator is an object which implements the iterator protocol.

The iterator protocol consists of two methods.

1. The iter() method, which must return the iterator object,
2. the next() method, which returns the next element from a sequence.

Iterators have several advantages:

• can work with infinite sequences
• save resources

Python has several built-in objects examples lists, tuples, strings, dictionaries or files, which implement the iterator protocol.

## Define Custom Iterator using Iterator Protocol.

class krange(object):
    
    def __init__(self, n):
        self.i = 0
        self.n = n
        
    def __iter__(self):
        return self
    
    def next(self):
        if self.i < self.n:
            i = self.i
            self.i += 1
            return i
        else:
            raise StopIteration()

k = krange(5)

print(k.next())
print(k.next())
print(k.next())
print(k.next())

0
1
2
3

Example: Write an iterator class reverse_iter, that takes a list and iterates it from the reverse direction.

class reverse_iter(object):
    
    def __init__(self, alist):
        self.alist = alist
        self.n = len(alist) -1 
        
    def __iter__(self):
        return self
    
    def next(self):
        if self.n > 0:
            item = self.alist[self.n]
            self.n -= 1
            return item
        else:
            raise StopIteration()
        

abc = reverse_iter([1,2,3,4,5])

print(abc.next())
print(abc.next())
print(abc.next())
print(abc.next())

5
4
3
2

Python Generators

Generator is a special routine that can be used to control the iteration behavior of a loop. A generator is similar to a function returning an array.
Generators are defined similar to a function and have the same properties like has parameters, can be called. But unlike functions, which return a whole array, a generator yields one value at a time. This requires less memory.

Generators in Python:

• Use the yield keyword and it May use several yield keywords
• Return an iterator

Generators simplifies creation of iterators. A generator is a function that produces a sequence of results instead of a single value.

So a generator is also an iterator. Here You don’t have to worry about the iterator protocol.

def gen(alist):
   for k in alist:
    yield k

g = gen([1,2,3,4,5])

print(next(g))
print(next(g))
print(g.__next__()) # g.next() in python2
print(g.__next__())

1
2
3
4

Can you think about how it is working internally?

When a generator function is called, it returns a generator object without even beginning execution of the function. When next method is called for the first time, the function starts executing until it reaches yield statement. The yielded value is returned by the next call.

Let’s check an example:

def foo():
    print("start Foo")
    for i in range(5):
        print(f"Before Yield i -- {i}")
        yield i
        print(f"After Yield i -- {i}")
    print("End Foo")

f = foo()
print(f.__next__())

start Foo
Before Yield i -- 0
0

print(f.__next__())

After Yield i -- 0
Before Yield i -- 1
1

print(f.__next__())

After Yield i -- 1
Before Yield i -- 2
2

print(f.__next__())

After Yield i -- 2
Before Yield i -- 3
3

print(f.__next__())

After Yield i -- 3
Before Yield i -- 4
4

print(f.__next__())

After Yield i -- 4
End Foo



---------------------------------------------------------------------------

StopIteration                             Traceback (most recent call last)

<ipython-input-72-2f4b6c2bfb23> in <module>
----> 1 print(f.__next__())


StopIteration: 

Python generator expression

A generator expression is created with round brackets.

aa = (a for a in range(1000)) # this return a generator
ab = [b for b in range(100)] # this return a list

print(next(aa))
print(next(aa))
print(next(aa))
print(next(aa))

print(next(aa))
print(next(aa))
print(next(aa))
print(next(aa))

0
1
2
3
4
5
6
7

Further Reading: http://www.dabeaz.com/generators-uk/