Menu

data engineering

Explore our insightful blog posts that delve deeper into the latest industry trends, best practices, and success stories. Gain valuable knowledge and stay informed about the ever-evolving landscape of digital transformation.

Suggested categories

Mastering Clojure Fundamentals for Beginners

1. Getting Started1.1. Running the Clojure InterpreterClojure is powerful, easy to use, general purpose programming language that runs on the JVM. Clojure like other Lisp dialects is dynamic, which means that it is not overly typed based system and gives you some room to breathe. In addition to that Clojure is compiled and can easily call upon the vast amount of Java libraries and the goodness of the JVM. I have created this series keeping in mind the average programmer who is more experienced in C-based programming languages like Java, Python, JavaScript. If you come from such programming languages and are looking into exploring the world of a full-fledged functional programming language, then Clojure is the one for you.This chapter is the starting of a series on Clojure. We will try to see the internals of Clojure and at the end of it seek to build a real world app.You can run Clojure both as a jar file or as an interpreter. We will look at how to distribute the Clojure code as a jar file later on. Right now lets first start Clojure as an interpreter. To do that install leiningen which is considered to be the easiest way to use Clojure.First, you need to install Java if you don’t have that installed already. The below instructions are for an Ubuntu machine. In case you have a different machine and are not able to install the dependencies, write to me and we will try to find what the issue might be. sudo apt-get update sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt install oracle-java8-installerview rawinstall_java8_ubuntu.bash hosted with ❤ by GitHubYou can then follow the installation steps as shown below. This is the same as the official installation steps which I followed in my Ubuntu machine. ➜ ~ wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein --2017-06-14 19:57:44-- https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.8.133 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.8.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 12871 (13K) [text/plain] Saving to: ‘lein’   lein 100%[==========================================================>] 12.57K --.-KB/s in 0.007s   2017-06-14 19:57:45 (1.72 MB/s) - ‘lein’ saved [12871/12871]   ➜ ~ sudo mv lein /usr/local/bin/lein [sudo] password for joy: ➜ ~ sudo chmod a+x /usr/local/bin/leinview rawinstall_leiningen.bash hosted with ❤ by GitHubOnce done you can start the lein repl command and this will download the necessary libraries and start a repl for you. ➜ ~ lein repl nREPL server started on port 37057 on host 127.0.0.1 - nrepl://127.0.0.1:37057 REPL-y 0.3.7, nREPL 0.2.12 Clojure 1.8.0 Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 Docs: (doc function-name-here) (find-doc "part-of-name-here") Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) Results: Stored in vars *1, *2, *3, an exception in *e   user=>view rawlein_repl.bash hosted with ❤ by GitHubNow check if you can run the code and if it gives the correct result.user=> (+ 1 1)2Voila! You have successfully started your Clojure interpreter and have run a Clojure program.1.2 AssignmentsThe use of global variables is recommended to avoid in Clojure and to keep the functions as clean and pure as possible. But you can create a persistent reference to a changing object. This is done through the use of Vars. user=> (def x 1) #'user/x user=> x 1 user=> (def y 2) #'user/y user=> (+ x y) 3 user=>view rawasignment.clj hosted with ❤ by GitHubVars are generally static. So, if you need to make it dynamic and have some calculations based on them, you will need to use the macro binding. Bindings are thread-local and do not propagate outside the thread. user=> (def ^:dynamic x 1) #'user/x user=> (def ^:dynamic y 1) #'user/y user=> (binding [x 2 y 3] (+ x y)) 5 user=> (+ x y) 2 user=>view rawdunamic_vars.clj hosted with ❤ by GitHub1.3 NumbersClojure is a dialect of Lisp and that means that means that operators are considered as functions and so come before the arguments that should go into the operation. user=> (+ 1 1) 2 user=> (* 2 3) 6 user=> (/ 17 5) 17/5 user=> ( float (/ 17 5)) 3.4view rawnumbers.clj hosted with ❤ by GitHub1.4 StringsStrings are believed to be one of the basic building blocks of programming. Almost everything that you do will involve manipulation of strings.In Clojure, a string literal is created by enclosing the string in double quotes.To concatenate some string variables you can use the `str` function: user=> (def a1 "My favorite lang is ") #'user/a1 user=> (def a2 "Clojure") #'user/a2 user=> (str a1 a2) "My favorite lang is Clojure" user=>view rawstrings.clj hosted with ❤ by GitHubYou can also use string formatting using the Format function. user=> (format "Any man whose errors take %s years to correct is quite the man." "10") "Any man whose errors take 10 years to correct is quite the man." user=>view rawstring_format.clj hosted with ❤ by GitHub1.5 FunctionsClojure is a functional programming language. So you will be creating a lot of functions. Functions that do something, functions that pass functions, functions that manipulate other functions and so on. You can consider a function to be unrealized functionality. Generally, functions are defined in the following manner.(defn name doc-string? attr-map? [params*] body)For example, a function that returns a square of a number can be created. (defn my_square "This is my my_square function" [x] (* x x))   (my_square 2)view rawsquare.clj hosted with ❤ by GitHubThe above code should return 4. Now the functions can be used in other expressions.(+ (my_square 2) (my_square 3))which should give the output 13.If you just need to create a small function that you can consider writing anonymous functions usingfn .user=> ((fn [x] (* x x)) 2)41.6 ConditionsOne way of implementing branching in Clojure is through the if statement which is basically a test. The format is basically:(if condition-statment <(optional)if condition is false return this>)The above construct will be clear in the below examples.user=> (if nil “truth value” “false value”)“false value”In Clojure, only nil and false are considered logical false.user=> (if “true” “truth value” “false value”)“truth value”But if cannot take multiple tests in the format of if..elif..else. For that you will need to use thecond macro. Then you can pass multiple test parameters. user=> (defn odd-even #_=> "Determines whether or not n is odd or even." #_=> [n] #_=> (cond #_=> (= (rem n 2) 0) "even" #_=> (= (rem n 2) 1) "odd")) #'user/odd-even user=>   user=> (odd-even 2) "even" user=> (odd-even 3) "odd"view rawmultiple_conditions.clj hosted with ❤ by GitHubThe above constructs are more in line with the imperative languages.Of course, you can convert any if constructs to filter, but more on that later.1.7 Data Containers and Data Structures.Clojure has four primary data structure or collections of data.1.Lists2.Vectors3.Maps4.SetsLists in Clojure are singly linked lists. So modifications of lists are efficient while random access is not. Lists are one of the most important data structures in Clojure. A code is data in Clojure and they are represented in lists.You can either use the list function or the quote literal to creating a list.user=> (list “clojure” “F#” “python” “java”)(“clojure” “F#” “python” “java”)user=> ‘(“clojure” “F#” “python” “java”)(“clojure” “F#” “python” “java”)Get the first or nth element of the list. user=> (def langs (list "clojure" "F#" "python" "java")) #'user/langs user=> (first langs) "clojure" user=> (nth langs 2) "python"view rawlist_lookup.clj hosted with ❤ by GitHubYou can treat the list like a stack with peek and pop. user=> (peek langs) "clojure" user=> langs ("clojure" "F#" "python" "java") user=> (pop langs) ("F#" "python" "java") user=>view rawas_stack.clj hosted with ❤ by GitHubYou can also extend a list to a new list with conj and concat. Notice that the conjoining in lists happen at the beginning of the list. user=> (conj langs "APL") ("APL" "clojure" "F#" "python" "java") user=> (conj langs '("OCAML" "erlang")) (("OCAML" "erlang") "clojure" "F#" "python" "java") user=> (concat langs '("OCAML" "erlang")) ("clojure" "F#" "python" "java" "OCAML" "erlang")view rawlist_manipulation.clj hosted with ❤ by GitHubB. VectorsVectors are similar to arrays in C and Java. The elements in a vector are indexed by contiguous integers. Vectors are optimized for random lookups.To create a vector you can use the vector function or just define them. user=> (vector "Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba") ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba"]   user=> ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba"] ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba"]view rawvector_defn.clj hosted with ❤ by GitHubRetrieving the nth element of the vector is optimised for a vector. user=> (def plants-in-lalbagh ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba"]) #'user/plants-in-lalbagh user=>   user=> (nth plants-in-lalbagh 2) "Ficus bengalensis var - krishnae" user=> (get plants-in-lalbagh 2) "Ficus bengalensis var - krishnae" user=>view rawvector_lookup.clj hosted with ❤ by GitHubConcatenation works the same way to lists but conjoining happens at the end of the vector. user=> (def plants-in-lalbagh ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba"]) #'user/plants-in-lalbagh user=>   user=> (nth plants-in-lalbagh 2) "Ficus bengalensis var - krishnae" user=> (get plants-in-lalbagh 2) "Ficus bengalensis var - krishnae" user=> (conj plants-in-lalbagh "eucalyptus") ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba" "eucalyptus"] user=> (def plants-in-lalbagh1 ["Amherstia Nobilis" "Adansonia digitata"]) #'user/plants-in-lalbagh1 user=> (def plants-in-lalbagh2 ["Araucaria Cookie" "Bombax Ceiba"]) #'user/plants-in-lalbagh2 user=> (concat plants-in-lalbagh) plants-in-lalbagh plants-in-lalbagh1 plants-in-lalbagh2 user=> (concat plants-in-lalbagh1 plants-in-lalbagh) plants-in-lalbagh plants-in-lalbagh1 plants-in-lalbagh2 user=> (concat plants-in-lalbagh1 plants-in-lalbagh2) ("Amherstia Nobilis" "Adansonia digitata" "Araucaria Cookie" "Bombax Ceiba") user=>view rawvector_characteristics.clj hosted with ❤ by GitHubC. Maps or HashMaps.Maps are equivalent to dictionaries in Python and HashMaps in Java. Maps associate keys to values and the keys must be comparable. There are numerous implementations of maps with various guarantees of ordering and different performance on the lookups.You can create a map using literals.user=> (def fav-filmmakers {:stanley-kubrik “26 July 1928”#_=> :satyajit-ray “2 May 1921”})#’user/fav-filmmakersuser=>Note that keywords are used as keys here. This is more of a convention and not really required. To access the value corresponding to a particular key, you can use the get function or use the key specified as a function. user=> ;; keys as a function.   user=> (:satyajit-ray fav-filmmakers) "2 May 1921" user=> ;; using the get function.   user=> (get fav-filmmakers :satyajit-ray) "2 May 1921"view rawmap_lookup.clj hosted with ❤ by GitHubD. SetsSets are similar to mathematical sets and are a collection of unique values. They are hashed and unordered.To create a set you can use a literal or the set function.user=> #{1 2 3 4 5}#{1 4 3 2 5}user=> (set [1 2 3 4 5])#{1 4 3 2 5}Sets are collections so you can count, conjugate them with other elements or break them or see if they contain some element. user=> (def banned-books #{"the hindus", "The God of Small Things"}) #'user/banned-books user=> (count banned-books) 2 user=> (seq banned-books) ("The God of Small Things" "the hindus") user=> (conj banned-books "Five Past Midnight in Bhopal") #{"The God of Small Things" "Five Past Midnight in Bhopal" "the hindus"} user=> (disj banned-books "The God of Small Things") #{"the hindus"} user=> (contains? banned-books "the hindus") trueview rawset_as_collections.clj hosted with ❤ by GitHubAs you can expect from the name, sets support set-operations like union, difference, intersection, etc. Set operations require the set namespace to be loaded. user=> (require '[clojure.set :as set]) niL user=> ; set operations user=> (set/union #{1 2} #{2 3} #{1 3 2} user=> (set/intersection #{1 2} #{2 3}) #{2} user=> (set/difference #{1 2} #{2 3}) #{1}view rawset_operations.clj hosted with ❤ by GitHubNote:DataStructures are functions:Maps are functions of their keysKeywords are functions of mapsSets are functions of their elementsVectors are functions of their indicesHope you liked this!To get regular updates on more content, learning material and general rants you can follow me on Twitter.

Learn More >

Abstract Base Classes in Python, Redefine Future of Banking

Python is a dynamic and interpreted language. The result is that we do not know the errors in the code unless we run them. This is not the case with languages which are more static and compiled.Say, you are writing a new banking API. The business case requirement is quite mission critical and so, the library needs to be future-proof. You want the future developers, who are going to use the API, to be implementing the functionality in a specific way.Let’s say, you have your Banking class which will implement some methods. While creating a class, you are kind of making a promise that this class will implement some methods. To keep things simple, we will only take the case of one method execute_payment.class Banking(object):""" Payment processor that does nothing, just logs """def __init__(self):self._logger = logging.getLogger(__name__)self._logger.setLevel(logging.INFO)def process_payment(self, destination_address, amount):""" Execute a payment to one receiving single addressreturn the transaction id or None """print("process payment for bank.")Similarly, you have started writing implementations for various banks.class BankingBank1(object): ...class BankingBank2(object): ...Now, you see a pattern in them. Being a firm believer in the DRY (Don’t Repeat Yourself) pattern, you will probably move the similar methods to a base class and the child class will implement specific functionality.But, the subclasses may not be implementing some necessary functionality. You put that on the TODO list and go on a vacation for some days. Once back, you see code like this.class Banking(object):""" Payment processor that does nothing, just logs """def __init__(self):self._logger = logging.getLogger(__name__)self._logger.setLevel(logging.INFO)def process_payment(self, destination_address, amount):""" Execute a payment to one receiving single addressreturn the transaction id or None """raise NotImplementedError("Not implemented yet")class BankingBank1(Banking):"""docstring for BankingBank1"""def __init__(self, arg):super(BankingBank1, self).__init__()self.arg = argdef process_payment(self, destination_address, amount):""" Execute a payment to one receiving single addressreturn the transaction id or None """print('process payment for bank 1')class BankingBank2(Banking):"""docstring for BankingBank1"""def __init__(self, arg):super(BankingBank2, self).__init__()self.arg = argdef user_action(bb_obj):if isinstance(bb_obj, BankingBank1):bb_obj.process_payment('destination_address', 100)elif isinstance(bb_obj, BankingBank2):print('Cannot process the payment.')else:print('implement this.')bb2 = BankingBank2(__name__)user_action(bb2)You are appalled. There are multiple if statements checking the type of the object and then deciding the course of action based on that. Classic LBYL. You make a mental note about educating everyone in the team why LBYL is bad. A champ of the EAFP principle, you refactor the code.class Banking(object): ...class BankingBank1(Banking): ...class BankingBank2(Banking): ...def user_action(bb_obj):try:bb_obj.process_payment('destination_address', 100)except NotImplementedError as e:print('Cannot process the payment.')bb2 = BankingBank2(__name__)user_action(bb2)But you are not happy. The issue with this kind of implementation is that the object gets created and error is not raised till the method is called. This might be a problem if some other methods are getting executed before this method is called. This is a problem in interpreted languages like Python. In compiled languages, this kind of error would get raised at the compilation phase itself.Things should fail fast and if possible at the instantiation level itself, isn’t it? There goes your mentor, the programming friend who always helps out and lo, he tells you that it can be achieved using the abstract base class and the abstract method decorator. He sketches out the rough basics and you start writing the code.""" Contains payment processors for executing payments """from abc import ABCMeta, abstractmethodclass Banking(metaclass=ABCMeta):""" Payment processor that does nothing, just logs """def __init__(self):self._logger = logging.getLogger(__name__)self._logger.setLevel(logging.INFO)@abstractmethoddef process_payment(self, destination_address, amount):""" Execute a payment to one receiving single addressreturn the transaction id or None """passclass BankingBank1(Banking):"""docstring for BankingBank1"""def __init__(self, arg):super(BankingBank1, self).__init__()self.arg = argdef process_payment(self, destination_address, amount):""" Execute a payment to one receiving single addressreturn the transaction id or None """print('process payment for bank 1')class BankingBank2(Banking):"""docstring for BankingBank1"""def __init__(self, arg):super(BankingBank2, self).__init__()self.arg = argtry:bb2 = BankingBank2(__name__)bb2.process_payment('destination_address', 100)except TypeError as e:print('Cannot process the payment.')You test if it works or not. Sure enough, TypeError gets raised while trying to create the object itself.You can now rest assure that no functionality will get implemented and the object will fail in the instantiation phase only.You know a new pattern now and want to know some more before calling it a day. You go through various resources:Collections.abc has some common patterns that can be implemented. You can create custom types using these patterns.Some good design patterns are implemented using abstract method.References:Leonardo Giordani. 2016. “Abstract Base Classes in Python”.Stackoverflow. “Why use abstract base classes in Python”Guido van Rossum. 2007. PEP 3119.Dan Bader. Abstract Base Classes in PythonHope you liked this!To get regular updates on more content, learning material and general rants you can follow me on Twitter.

Learn More >

Understanding Method Definition in Python

The idea of method or function definition is probably language and domain-agnostic, i.e., more of a rational way of writing reusable codes and a clever way of implementing DRY (Don’t Repeat Yourself)A method definition in Python is as easy as in any other language. The functional block starts with a def keyword, followed by a function name and parentheses.def hello_world():"""Print Hello World"""print "Hello World">>>hello_world()"Hello World"So, how does a method work in Python? We will first look into the instance methods. Let’s start with a familiar example.class Animal:"""Animal class"""def __init__(self, name):"""Initialize animal"""self.name = namedef animal_name(self):"""Return animal name"""return "This is a {}".format(self.name)Here, we have a class called Animal with two methods __init__ and animal_name, where former is a special method and later is a plain pythonic instance method declaration. It can be seen that we have passed ‘self’ keyword in both these methods. What does this mean? Let us try to understand it with an example.>>> Animal.animal_name<unbound method Animal.animal_name>>>>Animal.animal_name()Traceback (most recent call last): File "<stdin>", line 1, in <module> Animal.animal_name()TypeError: unbound method animal_name() must be called with Animal instance as first argument (got nothing instead)The keyword ‘self’ is just a naming convention (other words can also be used). By providing ‘self’ as the first argument we are specifying that the methods within the class Animal are bound to the class instance, i.e, the first argument passed is itself the instance of a class. The below example will make it clearer.>>>animal = Animal('dog')>>>animal.animal_name<bound method Animal.animal_name of <__main__.Animal instance at 0x0216F9E0>>>>>animal.animal_name()'This is a dog'In this case, we create an instance of class Animal which is passed as the first argument to the method ‘animal_name’ or more precisely speaking Bound method which is same as doing:>>>Animal.animal_name(Animal('cat'))'This is a cat'Bound and Unbound methodsIn simple terms, a bound method is the one which is dependent on the instance of the class as the first argument. That’s why they are called instance methods and in our example, animal_name is an instance method. Staticmethods are examples of unbound methods. Speaking in terms of our previous example we can find an analogy, any animal_name is first bound to Animal first and then supposed to display its characteristics.StaticmethodIn simpler terms, staticmethods are those methods which are not bound to the class instance and deal only with the arguments it takes. To declare a staticmethod we just have to add ‘@staticmethod’ decorator over the method definition as shown in the next example.class Animal:"""Animal class"""def __init__(self, name):"""Initialize Animal"""self.name = namedef animal_name(self):"""Return animal name"""return "This is a {}".format(self.name)@staticmethoddef message():"""Return message"""return "Love animals!"We can see in the example that we have not added ‘self’ keyword in the message which suggests that it is not related to the class or its instance. Since it is not related to class anyway, it is not mandatory to call staticmethods with instance of class. Following example will give a clear idea.>>>Animal.message()"Love animals!">>>a = Animal('dog')"Love animals!">>>Animal.message <function __main__.message>Hence, we see that staticmethods are not bound to the class. Also, the analogy with the example is that the message (“Love animals”) is more of a generic type and not related to the class, so staticmethods are the ones which do not relate to class anyway. By using staticmethods, we are essentially trying to demonstrate that the method depends on arguments and does not touch the instance (i.e. ‘self’). We can see the implementation in datetime module of CPython repository.ClassmethodMuch like staticmethods, classmethods also are independent of the instance of class but are dependent on the class itself, i.e., bound to the class. Thus, quite obviously it takes class as the first parameter. Classmethods can be created with ‘@classmethod’ decorator as described in the next example.class Animal:"""Animal class"""animal = 'dog'@classmethoddef change_animal(cls, new_animal):"""Change animal for class"""return cls.animal = new_animaldef print_animal(self):"""Return animal"""return self.animalIn the above example, we are passing ‘cls’ as the class reference. Let’s see how this works.>>>Animal.animal'dog'>>>Animal.change_animal<bound method classobj.change_nimal of <class __main__.Animal at 0x7f63aefca9a8>>>>>Animal.change_animal('bird')>>>Animal.animal'bird'The example makes it clear that classmethods are bound to the class itself and thus, is powerful enough to change the class properties; in this case, the class is variable. Also, it is notable that much like staticmethods, these do not need to be called on class objects. Hence, classmethods are also called factory methods or alternate constructors. An example of alternate constructor is shown below:class Animal:"""Animal class"""def __init__(self, legs):"""Initialize animal"""self.legs = legs@classmethoddef change_legs(cls):"""Change legs for animal."""return cls(4)>>>animal = Animal(2)>>>animal.legs2>>>animal = Animal.change_legs()>>>animal.legs4An implemented example of classmethod can be seen here. Apart from classmethods and staticmethods, abstract methods are of great usage. You can read about abstractmethods and abstract base classes here. Happy Python!

Learn More >

Demystifying Basic Statistics in Machine Learning

These days there is a Cambrian explosion of various data science and machine learning tools that make it very easy to start in machine learning. Probably, you are someone who has heard about the buzzword and wanted to try it out yourself. Maybe you have gone through tutorials on one of the hot and trending machine learning libraries such as scikit-learn and want to have an idea on how to implement machine learning. You recognize that you have all the prerequisites of a problem that make it suitable for machine learning. You have the data set and also a problem that seems to have a pattern to it, but you cannot pin it down using an algorithm. You threw the data set to the machine learning library and got something as an outcome.Now, starts the problem. You don’t understand what this ‘something’ is. More often than not this ‘something’ is a collection of numbers in a weird format. Apart from better interpretability, the business for which your product or insights are built, wants to know if this is the best that can be done. As you are the one who implemented this, they want you to prove that this is the state of the art results and if not, it’s you who needs to improve the results. And secretly you are starting to realize that you have no clue what is happening around you.Machine Learning is a buzzword right now and there are many who would be trying their hands at it. Therefore, it would help you a great deal if you are able to understand a little bit about the intricacies that make machine learning tools and libraries so powerful. In this post, we will look at the basic statistical concepts that make up the bedrock of analysis in machine learning.Types of DataUnderstanding different types of data will help you in choosing the different types of techniques that you may use to get insights into the data.The major types of Data are:Numerical:This represents some quantifiable thing that you can measure. This can be discrete, which is normally the counts of some event. As for example, how many times a person visits the same doctor when he is sick, or the number of clothes bought by a customer on an eCommerce platform. Another flavor is that the numerical data can be continuous as well. A characteristic of continuous data is that the range can be infinite. An example maybe, how much rainfall happened in a given year and so on.Categorical:These are data that has no inherent numerical meaning, such as man, woman. Or the state of birth of a group of people and so on. Generally, numbers are assigned to these values but the numbers themselves don’t mean anything. One of the good ways to denote categorical values is through graphs. So, in the figure, the scientists ‘Chandrasekhara Venkata Raman’ and ‘Jagadish Chandra Bose’ both have the birthplace as India which is shown by the edges.Ordinal: This is the mixture of numerical and categorical data. An example is the ratings given by a customer as in 5 stars is better than 1 star.Mean Median ModeThese are the measures of central tendency of a data set.sourceMean is given by the total of the values of the samples divided by the number of samples. In mathematical terms this would mean:where x̄ is the mean and there are n values of x. Σ means that you need to take the sum of all the data points.Below is an example where the mean of a set of data points is calculated.This is a simple example taken to explain discrete data. The mean can be calculated in case of continuous data as well.To calculate the median, sort the values and take the middle value. To illustrate let x denote the following numbers:x = [23, 40, 6, 74, 38, 1, 70]In this case, you would need to sort the list first and then take the midpoint to find the median.sorted_x = [1, 6, 23, 38, 40, 70, 74]In this case, the midpoint is 38 since the value 38 divides sorted x into 3 values which are smaller than 38 and three values which are higher than 38. Now, in case there are even number of values then the average of the two middle values are taken as the median. The advantage of the median over the mean is that median is less susceptible to outliers. So, in situations where there is a high chance that there may be outliers present in the data set, it is wiser to take the median instead of the mean. For example, to understand what is the per capita income of a country the median is taken because the rich may be extremely rich which would skew the average and show a different picture than what the average people might experience. In that case, probably, taking the median would give a better insight.Mode represents the most common value in a data set. Mode is most useful when you need to understand clustering or number of ‘hits’. For example, a retailer may want to understand the mode of sizes purchased so that he can set stocking labels optimally. Say, store A has a mode of ‘small’ while store B has a mode of ‘XXL’.In this post, we looked at some of the basic statistics that you will encounter while looking at data for machine learning. There are some more concepts that you need to know and so head over to my next post.Also, if you are interested in talking more on this, just drop me a message @alt227Joydeep. I would be glad to discuss this further. You can also hit the like button and follow me here, on Medium.References:Quora. Is statistics a prerequisite for machine learning.Understanding different data types.HackerEarth. Descriptive Statistics.Draw a skewed normal dist.

Learn More >

How can Intelligent Chatbots Make the Brand User Friendly?

Chatbots and Virtual Assistants are slowly making their way towards becoming the modern consumer’s first touchpoint towards interaction with a brand. From providers of information to reliable automation that assists consumers with their queries to become the front face of the company, Chatbots have evolved significantly over the years, making strides towards intelligence and self-governance.In addition to being automation capable of solving problems on a real-time basis, chatbots have also become an incredible tool in engaging and keeping a consumer interested in the brand. Unique chatbot experiences are sought and cherished by its users.There are a number of strategies that can be employed to significantly increase chatbot engagement. Two of the most common ways for going about this is to either create a sense of curiosity in the consumer or create touchpoints that surprise and delight the end-user. Both these ways significantly increase chatbot usage and often serve as the foundation on which other strategies are integrated. Here are a few other ways you can make your chatbots more interesting.Build Chatbots that learnA chatbot that is personalized to suit the needs of the user it is talking to is always preferred to a chatbot that seems distant. The trick here is to surprise and delight the consumers by allowing the chatbot to learn from its conversations and adopt a pattern that is more likable to the user. One of the best strategies you can adopt is measuring the retention rate of the chatbot. Since the chatbot learns, adapts, and reacts in certain ways to certain target groups, it is possible to analyze these chatbots with higher retention rates and obtain the factors that would allow you to create a unified and delightful experience for all your consumers.Remove MonotonyThe most common complaint that chatbots receive is about their repetitive and stale experience every time a consumer gets a chance to interact with them. Chatbots are expected to learn and improve to offer a more refined yet fresh experience every time the user comes in touch with them. A chatbot that is centered around ease and comfort with a pinch of fresh engagement can go a long way in removing the monotony that users often find themselves complaining about. Intermixing various interactive options such as gifs, emoticons, banter, and quips in between wait times, suggestions, or predictions of what the user might also like are strategic ways that can guarantee positive engagement. A chatbot’s flow of conversation must always inject a feeling of curiosity into the user’s minds and make them wonder what else said bot can do.Gamification of FunctionsGamification of your chatbot can be a powerful incentive if used in the right way. It also relies heavily on the purpose that your bot has been created to achieve and provide entertainment or information that might be of use to the consumer. If the main purpose of your chatbot is to provide information or instill habitual patterns to its users, then gamification might be an added bonus to go about it.Gamification can be as complex as a quiz based on the previous information to reap rewards such as virtual medals, certificates, or even gift coupons. They can be as simple as giving the user a celebratory GIF. Another good strategy would be the ‘claiming’ of daily rewards or the announcement of special, limited bonuses only available during a very short period of time. Other forms of gamification could include the display of a competitive scoring of friends, point-based level up, and a reward system. The goal here is to ultimately make the consumer keep coming back for more.EncouragementAnother brilliant strategy to increase engagement is to allow the chatbot to encourage the user’s to spend more time with it. This can be achieved through prompts and questions on the topic of conversation that would naturally pique the user’s curiosity. Chatbots that deliver seasonal, or daily content can end with a tease or short glimpse of the next day’s content to influence the return of the user or consumer. Chatbots can also suggest similar topics of conversation to the user to continue to engage them.HumanizationHumanizing the chatbot is one of the best ways to instill a sense of familiarity and trust with the users. A study by eMarketer revealed that respondents always preferred to communicate with a real person in a choice between a chatbot and the latter. Building a chatbot interface with human-like characteristics: name, display image would make the consumers feel like they are talking to a real person and would overall increase user engagement.Grab the User’s Attention with Push NotificationsA chatbot or virtual assistant can be a game-changing asset in building a user-friendly brand. But it can only be helpful if the visiting users notice and use it.Push Notifications, whether on mobile or desktop browsers will help grab the attention of users almost immediately. A popup hook accompanied by a notification sound may help redirect the user’s attention towards the chatbot and therefore prompt them into using it to help assist them better.

Learn More >

Deep Dive into Jupyter Notebooks to Master ML Analysis

In case you stumbled on my blog series, here’s a brief about my blogging activities. I make deep dives on the important machine learning algorithms. After completing an overview of machine learning I had started with linear regression. Now before moving on, I want to take a kind of break from the flow and discuss the tooling that an everyday machine learning engineer uses. In fact, I would be specifically talking about the most important tool that he/she would need in their day to day work: Jupyter Notebook. If you have been interested in machine learning at least for some time then I am sure you must have heard of this tool and actively using it. Through this post, I am hoping that I will be able to shed light on some interesting ways in which you can use jupyter notebooks to take your analysis to the next level.HistoryBut first, let’s look at the origin of this awesome software. In the data-science field as well as the broader scientific community, the kind of analysis that has been resorted to has been largely exploratory. Not only is the output important but the methodology and the way the output is generated is also important. Previously it was difficult to share the line of thinking, hence, there was always a need for such a tool which facilitates it. Therefore, the IPython Notebook was developed in 2011 by a team of researchers under Fernando Pérez and Brian Granger. It was then called IPython because the kernel could only execute Python code. In 2014 the Project Jupyter started as a spin-off project from the IPython project. In the new notebooks, the kernels were separate from the frontend and support for other languages apart from Python was brought in.Just as a side note, Jupyter is an acronym for Julia, Python and R, signifying that the project creators want to encompass the whole sky of machine learning under its umbrella and provide the power of lightning and flash to all its users.:)Version ControllingOnce you start maintaining jupyter notebooks for reproducible results, it is important to have an effective storing mechanism. Github is an excellent platform to store and view the notebooks as it renders the notebook when you open the respective file link on any web browser. There are two caveats though. First, is if you have a lot of JavaScript in your notebook, then they are not rendered in Github. You will then need to see them on your local machine. Other than that sometimes differences in code are tough to see and hence, the changes with the previous code are pretty hard to see due to the file structure of the ipynb files.The solution is to save python files with the jupyter notebook so that the differences in code can be tracked easily as well. This can be done by editing the config file:~/.ipython/profile_nbserver/ipython_notebook_config.py ### If you want to auto-save .html and .py versions of your notebook: # modified from: https://github.com/ipython/ipython/issues/8009 import os from subprocess import check_call   def post_save(model, os_path, contents_manager): """post-save hook for converting notebooks to .py scripts""" if model['type'] != 'notebook': return # only do this for notebooks d, fname = os.path.split(os_path) check_call(['ipython', 'nbconvert', '--to', 'script', fname], cwd=d) check_call(['ipython', 'nbconvert', '--to', 'html', fname], cwd=d)   c.FileContentsManager.post_save_hook = post_saveview rawipython_notebook_config.py hosted with ❤ by GitHubThis would save a .py and .html file with each notebook saved. Then the differences in the code can be easily shown in the console.Saving images as a separate file also has a lot of benefits as you can compare the differences between the two. Click on this link and see how the differences in the images can be easily shown.Two kinds of notebooksSince jupyter notebooks can be used as an exploratory tool as well as a final document that can be delivered and shown to the customer, it is highly advisable for analytics teams to have two kinds of folders for a specific use case. One can be named develop where exploratory analysis can be kept while the other can be named deliver to house the final notebooks. So you will have the following folder structure.– develop– ––+ analysis1.ipynb+ analysis1.py+ somefig.png– deliver–+ final_analysis.ipynb+ final_analysis.py+ fig1.pngMagicsThere are a lot of magics that can be used in jupyter notebooks. A listing of all the magics will make this post too long, hence, will only focus on the ones that I find most useful in my day to day work.The first one is using the magic bash. Bash commands make it easy to see the files in the current directory and also download files and other simple bash commands without leaving the comfort of the browser. You can also show the state of the directory where you are probably downloading some file and would like to show that the file is present in some remote directory like s3 and you will need to download the file.view rawbash_magics.ipynb hosted with ❤ by GitHubAnother one I like is showing math equations. Since we as data scientists work with a lot of algorithms, we need to articulate the algorithms that we use for other engineers who we are working with. Hence %%latex magic really helps us in that manner.view rawlatex.ipynb hosted with ❤ by GitHubStructuring a Jupyter NotebookIn the beginning, there should be the imports and the dependencies. For Python, imports should be ordered in the normal PEP8 way. i.e.,1.standard libraries2.related third-party imports3.local application/library/module importsNext, the version information should be given on at least the main dependencies like numpy, scipy, pandas etc.view rawstarting_notebook.ipynb hosted with ❤ by GitHubOnce you have reached near the end of your notebook you should consider two things.Is my notebook too big? In case its too big, then probably it will not be clear to the reader what the big picture is. Hence, probably you should consider breaking the notebook into separate notebooks. To do that you can take help of the FileLinks function.view rawlinking.ipynb hosted with ❤ by GitHubJupyter Notebooks are meant to be stories about data and hence should have a conclusion summarizing the main points and what has been discussed.DebuggingIf the code that you are trying to debug lands in the realm of pure python, then you can employ the age-old python debugging tools of pdb. The pdb commands and methods work in the jupyter notebook as well.view rawdebugging_using_python.ipynb hosted with ❤ by GitHubIn my experience pdb should help you a great deal while debugging your models. This is based on the assumptions that you create your models in python. In case you are creating your models in some other language then you will need to use the debugging tools for the respective language, but I am going to assume that jupyter would work the same way in case the language is an interpreted one like R, Julia, etc. In case you are finding some other kind of errors that look more like weird matrix issues, then probably the shape of the data that you are passing is wrong and you need to look closer at how you are building your matrices.ProfilingSince you will be dealing with huge amounts of data, it is a good thing to know such things as memory consumption, time spent in executing the cells, especially the cells that are doing most of the heavy lifting, and so on.%%time will tell you the amount of time needed to run the cell.view rawprofiling_time.ipynb hosted with ❤ by GitHubYou can also use line profiler or see how much memory the cells are taking. For that, you will need to install line_profiler and memory_profiler depending on your requirement. Then you can load the extensions.view rawprofiling_line_memory.ipynb hosted with ❤ by GitHub

Learn More >

Prolog Logic Programming: A Detailed Explanation

Prolog logic programming language is a general-purpose programming language. If you have never heard of this language and are a SQL aficionado, you will recognize the way of writing code in this language, which feels quite like a query. This is a declarative language or a rule-based language like SQL. Note that in “normal” languages that you encounter, like C, Java, and Python, you generally write out the ways in which you retrieve the data or do the computation. Here, you will just be defining what you want instead of writing out explicitly how to get it. Feels wonderful, isn’t it? Now, since this is a vast language by itself, only a minuscule part will be looked at.1.install swipl2.take a look at some basic examples to have some idea on how prolog code looks like3.ruminate on some basic ways in which we can reason about codeSWIPLSWI-Prolog is one of the most comprehensive implementation for prolog. It is also very widely used. You can use the SWI-Prolog interpreter to run the programs. To install the interpreter in ubuntu you can run the following commands.sudo add-apt-repository ppa:swi-prolog/stablesudo apt-get updatesudo apt-get install swi-prologNow, create a new hello.pl and type the below lines: /* * usage: swipl -q -f prolog_hello.pl -t main */ main :- write("Hello world!"), nl, fail.view rawprolog_hello.pl hosted with ❤ by GitHubYou have created a hello world program for prolog. In the terminal, run the following command to run this file and you should get the output in the terminal.➜ swipl -q -f hello.pl -t mainHello WorldSome code examplesTake a look at some code samples to understand the flow of logic in a prolog code. To show relationships between data, write in the following format.verb(noun1, noun2)which means that there is a relationship from noun1 to noun2 via relationship verb . Using this define the following relationships. has(jack,apples). % jack has apples. has(jack,plums). % jack has plums. has(ann,plums). % ann has plums. has(dan,money). % dan has money. fruit(apples). % apple is fruit. fruit(plums). % plums is fruit.view rawprolog1.pl hosted with ❤ by GitHubThis is logic programming where the facts and rules of the problem domain is defined. The following code will list the members of fruit class.https://gist.github.com/infinite-Joy/infinite-Joy/0cb2efac38507593cd0380e6d504e2aeThis is called a query. Save this to a listing.pl file and then execute the query. The command is written as:swipl -q -f -t➜ swipl -q -f listing.pl -t mainfruit(apples).fruit(plums).This shows a listing of all the relationships in the data.Similarly, based on the existing relationships, write the names of the people who have apples or plums. main :- has(X,apples), % who has apples has(Y,plums), % who has plums write(X), % write the apple lovers put(10), % new line write(Y), % write the plum lovers nl, halt. % new line and haltview rawprolog3.pl hosted with ❤ by GitHubRun the file in the terminal using the interpreter➜ prolog swipl -q -f listing1.pl -t mainjackannYou can find the whole file in this github gist.Now as an exercise can you find the people who have both apples and plums. Share the gist in the comments below.In case you found this interesting and would like to talk more on this, just drop me a message @alt227Joydeep. I would be glad to discuss this further. You can also hit the like button and follow me here, on Medium.

Learn More >

Graphs in Python and Prolog for Shortest Paths

Graphs are omnipresent in today’s computing world. From implementing compilers to computer games to maps, you will need to use graphs everywhere. In case you are a programmer and not sure what is a graph or any of the major graph algorithms, please take a look at these books on graph algorithms. In this blog, we will be talking about graphs in python and prologGraphs Let’s build a graph and see what the solution using standard techniques look like. First, let’s define a graph. We will define the graph in Python and try to visualize it as well.graph = {'a': {'w': 14, 'x': 7, 'y': 9},'b': {'w': 9, 'z': 6},'w': {'a': 14, 'b': 9, 'y': 2},'x': {'a': 7, 'y': 10, 'z': 15},'y': {'a': 9, 'w': 2, 'x': 10, 'z': 11},'z': {'b': 6, 'x': 15, 'y': 11}}This can be visualized in Python using the libraries networkx and matplotlib.%matplotlib inlineimport networkx as nximport numpy as npimport matplotlib.pyplot as pltimport pylab# initialise a directed graphG = nx.DiGraph()# add the edges and the weights.G.add_edges_from([('A', 'W')], weight=14)G.add_edges_from([('A', 'X')], weight=7)G.add_edges_from([('A', 'Y')], weight=9)G.add_edges_from([('B', 'W')], weight=9)G.add_edges_from([('B', 'Z')], weight=6)G.add_edges_from([('W', 'A')], weight=14)G.add_edges_from([('W', 'B')], weight=9)G.add_edges_from([('W', 'Y')], weight=2)G.add_edges_from([('X', 'A')], weight=7)G.add_edges_from([('X', 'Y')], weight=10)G.add_edges_from([('X', 'Z')], weight=15)G.add_edges_from([('Y', 'A')], weight=9)G.add_edges_from([('Y', 'W')], weight=2)G.add_edges_from([('Y', 'X')], weight=10)G.add_edges_from([('Y', 'Z')], weight=11)G.add_edges_from([('Z', 'B')], weight=6)G.add_edges_from([('Z', 'X')], weight=15)G.add_edges_from([('Z', 'Y')], weight=11)# value map so that we have different colors for different nodesval_map = {'A': 1.0,'B': 0.9,'W': 0.8,'X': 0.7,'Y': 0.6,'Z': 0.5}values = [val_map.get(node, 0.45) for node in G.nodes()]edge_labels=dict([((u,v,),d['weight'])for u,v,d in G.edges(data=True)])red_edges = [('A','Y'),('Y','W'), ('W', 'B')]edge_colors = ['black' if not edge in red_edges else 'red' for edge in G.edges()]node_labels = {node:node for node in G.nodes()}plt.axis('off')fig = plt.figure(1)fig = plt.figure(num=1, figsize=(10, 10), dpi=100)fig.set_size_inches(10.5, 10.5)# draw it out.pos=nx.spring_layout(G)nx.draw_networkx_edge_labels(G,pos,edge_labels=edge_labels)nx.draw_networkx_labels(G, pos, labels=node_labels)nx.draw(G,pos, node_color = values, node_size=1500,edge_color=edge_colors,edge_cmap=plt.cm.Reds)xmax = 1.2ymax = 1.1plt.xlim(-0.1, xmax)plt.ylim(-0.1, ymax)pylab.show()pylab.close()del figTo find the shortest path using python you probably will need to implement a known algorithm such as Dijkstra. You can try to look at the code linked here which I would argue is huge. The implementation will be the same in almost all object oriented and procedural code. But in Prolog, due to many things being handled by the compiler, the code gets reduced a lot.No need to write node classes or priority queuesJust write the relationships between the nodes and edges in a simple, clean manner.edge(a,w,14).edge(a,x,7).edge(a,y,9).edge(b,w,9).edge(b,z,6).edge(w,a,14).edge(w,b,9).edge(w,y,2).edge(x,a,7).edge(x,y,10).edge(x,z,15).edge(y,a,9).edge(y,w,2).edge(y,x,10).edge(y,z,11).edge(z,b,6).edge(z,x,15).edge(z,y,11).Now, looking at the code that crunches the numbers.findapath(X, Y, W, [X,Y], _) :- edge(X, Y, W). % findapath between X and Y has weight W  if there is an edge between X and Y of weight Wfindapath(X, Y, W, [X|P], V) :-                % else findapath between X and Y        \+ member(X, V),                       % is trueedge(X, Z, W1),                        % if we can findapath between X and Z with weight W1findapath(Z, Y, W2, P, [X|V]),         % and there is findapat between Z and Y of weight W2W is W1 + W2.                          % where W is W1 + W2:-dynamic(solution/2).findminpath(X, Y, W, P) :-\+ solution(_, _),findapath(X, Y, W1, P1, []),assertz(solution(W1, P1)),!, findminpath(X,Y,W,P).findminpath(X, Y, _, _) :-findapath(X, Y, W1, P1, []),solution(W2, P2),W1 < W2,retract(solution(W2, P2)),asserta(solution(W1, P1)), fail.findminpath(_, _, W, P) :- solution(W,P), retract(solution(W,P)).main :-    findminpath(a,b,W,P),    write('the minimum cost that we must pay.'),    write(W),    nl,    write('path that has the minimum cost.'),    write(P),    nl, halt.A huge shout-out to Rodney who has made a brilliant description of the above code.Last ThoughtsIn many cases involving graphs, Prolog helps us in writing clear and concise code. It is easier to express some data structures using Prolog that needs more careful code writing with intricate testing to come to the same solution.In case you found this interesting and would like to talk more on this, just drop me a message @alt227Joydeep. I would be glad to discuss this further. You can also hit the like button and follow me here, on Medium.References:Wikipedia. PrologPython Patterns — Implementing GraphsDjikstra algorithm implementationPython graph implementationFind path between two nodes in graph. Prolog discussion threadSO. create graphs using networkx

Learn More >

Principal Component Analysis and Dimension Reduction – Part II

In the previous post, we saw why we should be interested in Principal Component Analysis. In this post, we will do a deep dive and get to know how this is implemented.Now that you have some idea about how to change higher dimensions to lower dimensions, we will go through the below description which is shown in a jupyter notebook.I have downloaded the data of three companies that are in the Indian stock market from Quandl. We will try to understand the Indian ecosystem using this.These data are for the Hitech Corporation Ltd. Housing and Development Corporation Ltd. and Bhagyanagar India limited. The features of the data set are Date, Open pricing, the maximum price reached during the day, minimum price during the day, last price before closing, closing price, total traded quantity and turn over in lakhs.In order to load the data set, we will use the pandas library. If you are interested, in the dataset, you can create an account on the website which is free and download the data sets. import pandas as pd df_hitech = pd.read_csv("NSE-HITECHCORP.csv") df_hitech.head()view rawpca1.py hosted with ❤ by GitHub df_hudco = pd.read_csv("NSE-HUDCO.csv") df_hudco.head()view rawpca2.py hosted with ❤ by GitHubTaking a look at the shape of the matrices. print(df_hitech.shape, df_bhagyanar.shape, df_hudco.shape)view rawpca3.py hosted with ❤ by GitHub((51, 8), (55, 8), (53, 8))So our three data sets are three matrices of dimensions (48 x 8), (52 x 8) and (50 x 8) where the columns are the different features and every row represents a separate data sample.VisualizationTo get the feeling of how the different stocks are distributed along opening stock prices and total trade quantities, let us visualize them using histograms.Get the relevant data. Right now we are only interested in visualization and as we cannot show all the features in a plot, we will just focus on some specific columns. import numpy as np from bokeh.charts import output_file, show, Line from bokeh.charts import Area, show, TimeSeries from bokeh.io import output_notebook   X1 = df_hitech.loc[0:5, ['Date', 'Open', 'Total Trade Quantity']].values X2 = df_bhagyanar.loc[0:5,['Date', 'Open', 'Total Trade Quantity']].values X3 = df_hudco.loc[0:5, ['Date', 'Open', 'Total Trade Quantity']].values   # Get a feel for the data. print([x for x in X1]) print([x for x in X2])view rawpca4.py hosted with ❤ by GitHub[array([‘2017-08-02’, 209.1, 2935.0], dtype=object), array([‘2017-08-01’, 212.0, 5094.0], dtype=object), array([‘2017-07-31’, 212.0, 6803.0], dtype=object), array([‘2017-07-28’, 208.0, 2023.0], dtype=object), array([‘2017-07-27’, 213.05, 6714.0], dtype=object), array([‘2017-07-26’, 218.95, 3389.0], dtype=object)][array([‘2017-08-02’, 24.5, 5857.0], dtype=object), array([‘2017-08-01’, 24.75, 12622.0], dtype=object), array([‘2017-07-31’, 25.05, 31276.0], dtype=object), array([‘2017-07-28’, 25.15, 11483.0], dtype=object), array([‘2017-07-27’, 26.0, 19290.0], dtype=object), array([‘2017-07-26’, 26.5, 17808.0], dtype=object)]Show how the data is distributed for one company. data = dict( hitech=[x[1] for x in X1], bhagyagar=[x[1] for x in X2], date=[x[0] for x in X1], )   line_plot = Line(data, x='date', title="Area Chart", legend="top_left", xlabel='time', ylabel='memory') show(line_plot)view rawpca5.py hosted with ❤ by GitHub StandardizingWe want to compare oranges to oranges and apples to apples. Generally, we are more interested in how the changes are happening and not too much concerned about the scale on which the data set was published. In this case, you can see the data set is comprised of numbers that were gathered on various scales, so it would be wise to standardize the data set to the unit scale with mean as 0 and variance as 1. from sklearn.preprocessing import StandardScaler   column_names = ['Open', 'High', 'Low', 'Last', 'Close', 'Total Trade Quantity', 'Turnover (Lacs)']   X_hitech = df_hitech.loc[:, column_names].values X_bhagyanagar = df_bhagyanar.loc[:, column_names].values X_hudco = df_hudco.loc[:, column_names].values   X = np.concatenate([X_hitech, X_bhagyanagar, X_hudco], axis=0)   X_std = StandardScaler().fit_transform(X)   print(X_std.shape)view rawpca6.py hosted with ❤ by GitHub(159, 7)Covariance matrixWe can perform the Eigen decomposition on the covariance matrix.In terms of code, this can be represented using the below format.Please note that the shape of the standardized matrix is 150 x 7. So we should get a covariance matrix of 7 x 7. mean_vec = np.mean(X_std, axis=0)   cov_mat = (X_std - mean_vec).T.dot((X_std - mean_vec)) / (X_std.shape[0]-1)   print('Covariance matrix \n{}'.format(cov_mat)) print('Covariance matrix shape is: {}'.format(cov_mat.shape))view rawpca7.py hosted with ❤ by GitHubCovariance matrix[[ 1.00632911 1.00550536 1.00597911 1.00542205 1.00545316 -0.06315605-0.05901863][ 1.00550536 1.00632911 1.00541311 1.00598897 1.00602285 -0.0542054-0.0497978 ][ 1.00597911 1.00541311 1.00632911 1.00562847 1.00565687 -0.06570958-0.06198619][ 1.00542205 1.00598897 1.00562847 1.00632911 1.00628689 -0.06051605-0.05646143][ 1.00545316 1.00602285 1.00565687 1.00628689 1.00632911 -0.05997139-0.05587568][-0.06315605 -0.0542054 -0.06570958 -0.06051605 -0.05997139 1.006329110.99963087][-0.05901863 -0.0497978 -0.06198619 -0.05646143 -0.05587568 0.999630871.00632911]]Covariance matrix shape is: (7, 7)We showed the code just to show how the covariance works. In general, the cov function is used. covariance_matrix = np.cov(X_std.T) print(covariance_matrix)view rawpca8.py hosted with ❤ by GitHub[[ 1.00632911 1.00550536 1.00597911 1.00542205 1.00545316 -0.06315605-0.05901863][ 1.00550536 1.00632911 1.00541311 1.00598897 1.00602285 -0.0542054-0.0497978 ][ 1.00597911 1.00541311 1.00632911 1.00562847 1.00565687 -0.06570958-0.06198619][ 1.00542205 1.00598897 1.00562847 1.00632911 1.00628689 -0.06051605-0.05646143][ 1.00545316 1.00602285 1.00565687 1.00628689 1.00632911 -0.05997139-0.05587568][-0.06315605 -0.0542054 -0.06570958 -0.06051605 -0.05997139 1.006329110.99963087][-0.05901863 -0.0497978 -0.06198619 -0.05646143 -0.05587568 0.999630871.00632911]]Next is to perform an eigen decomposition on the above matrix. Eigen decomposition is the method where we decompose a square matrix into its Eigen vectors and Eigen values. The Eigen vectors are basically hyperplanes to which the data gets projected into. Take a look at wikipedia to learn more.We find the Eigen vectors and the Eigen values using the numpy library. In PCA the Eigen vectors are called loadings. Let’s call them the matrix W. eigen_vals, eigen_vecs = np.linalg.eig(covariance_matrix)   print(eigen_vals) print(eigen_vecs)view rawpca9.py hosted with ❤ by GitHub[ 5.04062300e+00 1.99469506e+00 6.70989466e-03 1.52221819e-035.08344416e-04 2.04592597e-04 4.06913481e-05][[-0.44636953 -0.0255697 -0.00233638 0.61091357 0.39241169 -0.522008190.02000256][-0.44626454 -0.03201096 -0.03801748 -0.31356635 0.66065443 0.51116882-0.04791904][-0.44644488 -0.02362002 0.04386902 0.47716468 -0.50791923 0.55804311-0.03353175][-0.44641063 -0.02742468 0.0018537 -0.3959805 -0.28922772 -0.3236239-0.67437107][-0.44640923 -0.02782594 -0.0024592 -0.37808739 -0.25605417 -0.223672430.73579569][ 0.04405545 -0.70563431 0.70623273 -0.01919751 0.03123937 -0.005181540.0018351 ][ 0.04224484 -0.705916 -0.70558526 0.02560183 -0.03707468 0.00362514-0.00206853]]To know why the Eigen values and Eigen vectors are important, take a look at the math below. If this sounds too heady, you can skip this section and just focus on the code.So, we know that after performing the Eigen decomposition we get the Eigen values and the Eigen vectors. Let’s call them λ and W. Therefore, in terms of principal component analysis, we will say that the scores are the product of matrices X and W, i.e.,assuming T is the score.Singular Value DecompositionFinding the Eigen values and Eigen vectors are computationally not the most efficient way of finding the loadings or the matrix W. We have a shortcut to do the same thing and that is through Singular Value Decomposition or SVD.How it works is that the same input matrix X can be broken down to three matrices U, Σ, and V*.Let us call this equation 1.U is the left singular vector and V* is the right singular vector. So Σ will contain singular values in its diagonal and they are ordered. Please note that although V* means conjugate of V. Since we will be dealing with real world data, you can think of this as the transpose of V.Now, interestingly the matrix V turns out to be identical to the loadings or W.How is this transformation beneficial to us. Let’s explore that below. We know that the scores T can be shown with the given equationComing from equation 1 above we can write –Since V*V is the Identity Matrix, and W is the same as V, we getSo we can find the scores if we can compute U and Σ, which can be easily done. If you are interested in trying this right away, take a look at this SVD computation example.Selecting Principal ComponentsNow we will select the principal components. This is mostly based on human judgment but many thumb rules can be followed. Understanding of how many components you incorporate will ultimately come through experience and experimentation.Right now with the help of the Eigen decomposition we transformed our data to the new set of planes with the eigen vectors as the axes.We need to decide which Eigen vectors can be dropped without losing too much information. Please note that the Eigen values with the least amount of value are the ones that have the least amount of information regarding the data distribution. Why is that so? Remember the definition of Eigen vectors and Eigen values.The eigen value λ is the amount the vectors elongate or shrink. So, if the value of λ is small that would mean that the effect ν has on matrix A will be small.We will try to see how much each value contributes to the overall variance.We will try to see how much each value contributes to the overall variance. total = sum(eigen_vals) distribution = [100 * (i/total) for i in sorted(eigen_vals, reverse=True)]   cumulative_distribution = [] sum = 0 for i in distribution: sum += i cumulative_distribution.append(sum)   bottom = [0] + cumulative_distribution[:-1] left = [2*x for x in range(7)] right = map(lambda x: x+1, left) print(cumulative_distribution) print(len(cumulative_distribution)) print(bottom) print(len(bottom)) print(list(left)) print(list(right))view rawpca12.py hosted with ❤ by GitHub[71.556013843924504, 99.872439612636256, 99.967692385037282, 99.989301591992117, 99.996517981727536, 99.999422351033218, 99.999999999999986]7[0, 71.556013843924504, 99.872439612636256, 99.967692385037282, 99.989301591992117, 99.996517981727536, 99.999422351033218]7[0, 2, 4, 6, 8, 10, 12][1, 3, 5, 7, 9, 11, 13] from bokeh.plotting import figure, show   p = figure(width=400, height=400) p.quad(top=cumulative_distribution, bottom=bottom, left=[1, 2, 3, 4, 5, 6, 7], right=[1.2, 2.2, 3.2, 4.2, 5.2, 6.2, 7.2], color="#B3DE69")   show(p)view rawpca13.py hosted with ❤ by GitHubYou can see that after the first two components the contribution of the remaining to the total is almost nonexistent and cannot be seen at all. They are there and you can zoom in to see the others. The first two principal components can explain more than 99% of the data that we have.Now, since you know where the majority of the data in your data set is coming from it becomes obvious to trim those extra dimensions. Effectively we will construct a projection matrix to transform our data to a new feature subspace. In this case, we will trim all the other features and take into account only two features. So we are reducing our 7-dimensional feature space to 2-dimensional feature subspace. This is possible since we are only choosing the top 2 Eigen vectors with the highest Eigen values to construct our (150 x 2) Eigen vector matrix W. # Mapping eigen value to eigen vector eigen_pairs = [(np.abs(eigen_vals[i]), eigen_vecs[:,i]) for i in range(len(eigen_vals))]   # Sort the (eigenvalue, eigenvector) tuples from high to low eigen_pairs.sort() eigen_pairs.reverse()   # check if the value is decreasing print([e[0] for e in eigen_pairs])view rawpca14.py hosted with ❤ by GitHub[5.0406230005245538, 1.9946950557326693, 0.0067098946634397017, 0.0015222181861223813, 0.00050834441617230038, 0.00020459259729253709, 4.0691348102287684e-05]https://gist.github.com/infinite-Joy/91177e07b9916e2f268e33a4154c6360/[[-0.44636953 -0.0255697 ][-0.44626454 -0.03201096][-0.44644488 -0.02362002][-0.44641063 -0.02742468][-0.44640923 -0.02782594][ 0.04405545 -0.70563431][ 0.04224484 -0.705916 ]]Dimensionality ReductionUsing this new loading matrix W, we can reduce the dimensions of the original matrix X. This can be done using the matrix multiplication property, whereby if you multiply two matrices of dimensions (m x n) and (n x p), you get a new matrix of dimensions (m x p). So, in this case, we multiply matrix X, which is (159 x 7) and matrix W, which is (7 x 2) to get a new matrix (159 x 2).Taking a look at the code below. Y = X_std.dot(matrix_w)   print(Y[:4])view rawpca16.py hosted with ❤ by GitHubarray([[-3.12570525, 0.1778062 ],[-3.22460186, 0.17146108],[-3.26460636, 0.16888734],[-3.227314 , 0.17157395]])If you remember, the shapes of hitech, bhagyanagar and hudco were (51, 8), (55, 8), (53, 8). Hence, to create the mappings, we create a Z vector/list with shape (159, 1).https://gist.github.com/infinite-Joy/infinite-Joy/446130b2e02e914359ecca1e5610916dYou can see that two companies are very tightly packed, saying that there is not much variation in them while the third company has a lot of variation. This company has had a rocky year.In productionThe code that is shown above is mainly for learning purposes. In production, we will be using the inbuilt function from the sklearn library. from sklearn.decomposition import PCA as productionPCA prod_pca = productionPCA(n_components=2) Y_prod = prod_pca.fit_transform(X_std)   print(Y_prod[:2])   print(Y[:2])view rawpca18.py hosted with ❤ by GitHubThe graphs are opposite as the matrices are reversed in their sign. In case you are implementing this in production, you may not be finding the eigen values to understand if you are capturing the majority of the variance in the data set, as was done earlier. So, to find if your new feature subspace is capturing the intended amount of variance, you can use the explained_variance_ratio_ attribute. print(prod_pca.explained_variance_) print(prod_pca.explained_variance_ratio_) print(np.sum(prod_pca.explained_variance_ratio_.tolist()))view rawpca20.py hosted with ❤ by GitHub[ 5.00892097 1.9821498 ][ 0.71556014 0.28316426]0.99872439612636232Generally, people see the explained variance ratios or keep a hard cutoff threshold of maybe 95%. In our case, the cumulative graph, generated by pca13.py was getting most of the variation and the inflection shoulder was quite steep. But even in cases where the shoulder is quite shallow, having such a cutoff will enable you to have a principled and consistent way to understand your data.You can also download the whole blog as a jupyter notebook if you want to try out the code yourself.Now that you know about Principal Component Analysis take a look at the decision systems around you, where data is the primary input and try to observe if they are using more data than necessary. If not, there is your gold mine. References:Siral Raval. Dimensionality Reduction. YouTubeImportance of Principal Component Analysis. QuoraPrincipal Component Analysis.Washington University.YouTubeMinimizing reconstruction error is equivalent to maximizing variance.Statistics StackExchangeMaking sense of Eigen Vectors and Eigen Values and PCA – stats.stackexchange.complotly.Principal Component Analysis.

Learn More >