English Freelance Jobs

A chatbot is an artificial intelligence powered piece of software in a device, application, web site or alternative networks that try to complete consumer’s needs and then assist them to perform a selected task. Now a days almost every company has a chatbot deployed to interact with the users.

Chatbots are often used in many departments, businesses and every environment. They are artificial narrow intelligence (ANI). Chatbots only do a restricted quantity of task i.e. as per their design. However, these Chatbots make our lives easier and convenient. The trend of Chatbots is growing rapidly between businesses and entrepreneurs, and are willing to bring chatbots to their sites. You might also produce it yourself using Python.

How do chatbots work?

There are broadly two variants of chatbots: Rule-Based and Self learning.

1. In a Rule-based approach, a bot answers questions based on some rules on that it is trained on. The rules outlined could be very easy to very complicated. The bots will handle easy queries but fail to manage complicated ones.
2. The Self learning bots are those that use some Machine Learning-based approaches and are positively a lot of economical than rule-based bots. These bots may be of additional two types: Retrieval based or Generative.
   1. In retrieval-based models, Chatbot uses the message and context of conversation for selecting the best response from a predefined list of bot messages.
   2. Generative bots can generate the answers and not always reply with one of the answers from a set of answers. This makes them more intelligent as they take word by word from the query and generates the answers.

Building a chatbot using Python

NLP:

The field of study that focuses on the interactions between human language and computers is called Natural Language Processing. NLP is a way for computers to analyze, understand, and derive meaning from human language in a smart and useful way. However, if you are new to NLP, you can read Natural Language Processing in Python.

NLTK:

NLTK (Natural Language Toolkit) is a leading platform for building Python programs to work with human language data. It provides easy-to-use lexical resources such as WordNet, along with a suite of text processing libraries.

Importing necessary libraries

import nltk 

import numpy as np 

import random 

import string # to process standard python strings

Copy the content in text file named ‘chatbot.txt’, read in the text file and convert the entire file content into a list of sentences and a list of words for further pre-processing.

f=open('chatbot.txt','r',errors = 'ignore')

raw=f.read()

raw=raw.lower()# converts to lowercase

nltk.download('punkt') # first-time use only

nltk.download('wordnet') # first-time use only

sent_tokens = nltk.sent_tokenize(raw)# converts to list of sentences 

word_tokens = nltk.word_tokenize(raw)# converts to list of words

Pre-processing the raw text

We shall now define a function called LemTokens which will take as input the tokens and return normalized tokens.

lemmer = nltk.stem.WordNetLemmatizer()

#WordNet is a semantically-oriented dictionary of English included in NLTK.

def LemTokens(tokens):     

return [lemmer.lemmatize(token) for token in tokens]

remove_punct_dict = dict((ord(punct), None) for punct in string.punctuation) 

def LemNormalize(text):     

return LemTokens(nltk.word_tokenize(text.lower().translate(remove_punct_dict)))

Keyword matching

Define a function for greeting by bot i.e. if user’s input is greeting, the bot shall return a greeting response.

GREETING_INPUTS = ("hello", "hi", "greetings", "sup", "what's up","hey",)

GREETING_RESPONSES = ["hi", "hey", "*nods*", "hi there", "hello", "I am glad! You are talking to me"]

def greeting(sentence):

for word in sentence.split():

if word.lower() in GREETING_INPUTS:

return random.choice(GREETING_RESPONSES)

Generate responses

To generate a response from our bot for input queries, the concept of document similarity is used. Therefore, we start by importing necessary modules.

From scikit learn library, import the TFidf vector to convert a collection of raw documents to a matrix of TF-IDF features

from sklearn.feature_extraction.text import TfidfVectorizer

Also, import cosine similarity module from scikit learn library

from sklearn.metrics.pairwise import cosine_similarity

This will be used to find the similarity between words entered by the user and therefore the words within the corpus. This can be the simplest possible implementation of a chatbot.

Define a function response that searches the user’s vocalization for one or more known keywords and returns one of several possible responses. If it doesn’t find the input matching any of the keywords, it returns a response: “I’m sorry! I don’t understand you”

def response(user_response):

robo_response=''

sent_tokens.append(user_response)

TfidfVec = TfidfVectorizer(tokenizer=LemNormalize, stop_words='english')

tfidf = TfidfVec.fit_transform(sent_tokens)

vals = cosine_similarity(tfidf[-1], tfidf)

idx=vals.argsort()[0][-2]

flat = vals.flatten()

flat.sort()

req_tfidf = flat[-2]

if(req_tfidf==0):

robo_response=robo_response+"I am sorry! I don't understand you"

return robo_response

else:  robo_response = robo_response+sent_tokens[idx]

return robo_response

I have tried to explain in simple steps how you can build your own chatbot using NLTK and of course it’s not an intelligent one.

I hope you guys have enjoyed reading.

Happy Learning!!!

NLP is a branch of data science that consists of systematic processes for analyzing, understanding, and deriving information from the text information in a smart and efficient manner. By utilizing NLP and its parts, one can organize the massive chunks of text information, perform various automated tasks and solve a wide range of issues like – automatic summarization, machine translation, named entity recognition, relationship extraction, sentiment analysis, speech recognition, and topic segmentation etc.

NLTK (Natural Language Toolkit) is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to lexical resources like WordNet, along with a collection of text processing libraries for classification, tokenization, stemming, and tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries.

NLTK has been called “a wonderful tool for teaching and working in, computational linguistics using Python,” and “an amazing library to play with natural language.”

Downloading and installing NLTK

1. Install NLTK: run pip install nltk
2. Test installation: run python then type import nltk and run nltk.download() and download all packages.

Pre-Processing with NLTK

The main issue with text data is that it's all in text format. However, the Machine learning algorithms need some variety of numerical feature vector so as to perform the task. Thus before we have a tendency to begin with any NLP project we'd like to pre-process it to form it ideal for working. Basic text pre-processing includes:

• Converting the whole text into uppercase or lowercase, in order that the algorithm doesn't treat the same words completely different in several cases.
• Tokenization: Process of converting the normal text strings into a list of tokens i.e. words that we actually want. The NLTK data package includes a pre-trained Punkt tokenizer for English.

           import nltk

           from nltk.tokenize import word_tokenize

           text = "God is Great! I won a lottery."

           print(word_tokenize(text))

           Output: ['God', 'is', 'Great', '!', 'I', 'won', 'a', 'lottery', '.']

• Noise removal: Process of removing everything that isn’t in a standard number or letter.
• Stop word removal: A stop word is a commonly used word (such as “the”, “a”, “an”, “in”). We would not want these words or taking up valuable processing time. For this, we can remove them easily, by storing a list of words that you consider to be stop words. NLTK (Natural Language Toolkit) in python has a list of stopwords stored in sixteen different languages. You can find them in the nltk_data directory.  home/Saad/nltk_data/corpora/stopwords is the directory address.

           import nltk

           from nltk.corpus import stopwords

           set(stopwords.words('english'))

• Stemming: Stemming is the process of reducing the words to its root form. Example if we were to stem the following words: “Connects”, “Connecting”, “Connected”, “and Connection”, the result would be a single word “Connect”.

           # import these modules

           from nltk.stem import PorterStemmer

           from nltk.tokenize import word_tokenize   

           ps = PorterStemmer()  

           # choose some words to be stemmed

           words = ["Connect", "Connects", “Connected”, "Connecting", "Connection", "Connections"]

           for w in words:

           print(w, " : ", ps.stem(w)) 

• Lemmatization: Lemmatization is the process of grouping along the various inflected forms of a word in order that they may be analyzed as a single item. Lemmatization is similar to stemming but it brings context to the words. Therefore it links words with similar meaning to one word.

           # import these modules

           from nltk.stem import WordNetLemmatizer  

           lemmatizer = WordNetLemmatizer()  

           print("rocks :", lemmatizer.lemmatize("rocks"))

           print("corpora :", lemmatizer.lemmatize("corpora"))  

           # a denotes adjective in "pos"

          print("better :", lemmatizer.lemmatize("better", pos ="a"))

          -> rocks : rock

          -> corpora : corpus

          -> better : good

Now we need to transform text into a meaningful vector array. This vector array is a representation of text that describes the occurrence of words within a document. For example, if our dictionary contains the words {Learning, is, the, not, great}, and we want to vectorize the text “Learning is great”, we would have the following vector: (1, 1, 0, 0, 1). A problem is that extremely frequent words begin to dominate within the document (e.g. larger score), however might not contain as much informational content. Also, it will offer additional weight to longer documents than shorter documents.

One approach is to rescale the frequency of words or the scores for frequent words called Term Frequency-Inverse Document Frequency.

• Term Frequency: is a scoring of the frequency of the word in the current document.

           TF = (Number of times term t appears in a document)/ (Number of terms in the document)

• Inverse Document Frequency: It is a scoring of how rare the word is across documents.

           IDF = 1+log(N/n), where, N is the number of documents and n is the number of documents a term t has appeared in.

           Tf-idf weight is a weight often used in information retrieval and text mining.

           Tf-IDF can be implemented in scikit learn as:

           from sklearn.feature_extraction.text import TfidfVectorizer

           corpus = [

           ...     'This is the first document.’

           ...     'This document is the second document.’

           ...     'And this is the third one.’

           ...     'Is this the first document?',]

           >>> vectorizer = TfidfVectorizer()

           >>> X = vectorizer.fit_transform(corpus)

           >>> print(vectorizer.get_feature_names())

           ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']

           >>> print(X.shape)

           (4, 9)

• Cosine similarity: TF-IDF is a transformation applied to texts to get two real-valued vectors in vector space. We can then obtain the Cosine similarity of any pair of vectors by taking their dot product and dividing that by the product of their norms. That yields the cosine of the angle between the vectors. Cosine similarity is a measure of similarity between two non-zero vectors.

           Cosine Similarity (d1, d2) =  Dot product(d1, d2) / ||d1|| * ||d2||

          import numpy as np

          from sklearn.metrics.pairwise import cosine_similarity

          # vectors

          a = np.array([1,2,3])

          b = np.array([1,1,4])

          # manually compute cosine similarity

          dot = np.dot(a, b)

          norma = np.linalg.norm(a)

          normb = np.linalg.norm(b)

          cos = dot / (norma * normb)

After completion of cosine similarity matric we perform algorithmic operation on it for Document similarity calculation, sentiment analysis, topic segmentation etc.

I have done my best to make the article simple and interesting for you, hope you found it useful and interesting too.

The importance of extracting information from the web is becoming increasingly loud and clear. Every few weeks, I realize myself in a situation where we need to extract information from the web to create a machine learning model. We have to pull or extract a large amount of information from websites and we would like to do it as quickly as possible. How would we do it without manually going to every web site and getting the data? Web Scraping simply makes this job easier and faster.

Why is web scraping needed?

Web scraping is used to collect large information from websites. But why does someone have to collect such large data from websites? Let’s look at the applications of web scraping: 

1. Price Comparison: Services such as ParseHub use web scraping to collect data from online shopping websites and use it to compare the prices of products.
2. Social Media Scraping: Web scraping is used to collect data from Social Media websites such as Twitter to find out what’s trending.
3. Email address gathering: Many companies that use email as a medium for marketing, use web scraping to collect email ID and then send bulk emails.
4. Research and Development: Web scraping is used to collect a large set of data (Statistics, General Information, Temperature, etc.) from websites, which are analyzed and used to carry out Surveys or for R&D.
5. Job listings: Details regarding job openings, interviews are collected from different websites and then listed in one place so that it is easily accessible to the user.

Web scraping is an automated method used to extract large amounts of data from websites. The data on the websites are unstructured. Web scraping helps collect these unstructured data and store it in a structured form. There are different ways to scrape websites such as online Services, APIs or writing your own code.

Why Python is best for Web Scraping

Features of Python which makes it more suitable for web scraping:

1. Ease of Use: Python is simple to code. You do not have to add semi-colons “;” or curly-braces “{}” anywhere. This makes it less messy and easy to use.
2. Large Collection of Libraries: Python has a huge collection of libraries such as Numpy, Matlplotlib, Pandas etc., which provides methods and services for various purposes. Hence, it is suitable for web scraping and for further manipulation of extracted data.
3. Dynamically typed: In Python, you don’t have to define datatypes for variables, you can directly use the variables wherever required. This saves time and makes your job faster.
4. Easily Understandable Syntax: Python syntax is easily understandable mainly because reading a Python code is very similar to reading a statement in English. It is expressive and easily readable, and the indentation used in Python also helps the user to differentiate between different scope/blocks in the code.
5. Small code, large task: Web scraping is used to save time. But what’s the use if you spend more time writing the code? Well, you don’t have to. In Python, you can write small codes to do large tasks. Hence, you save time even while writing the code.
6. Community: What if you get stuck while writing the code? You don’t have to worry. Python community has one of the biggest and most active communities, where you can seek help from.

How does web scraping work

To extract data using web scraping with python, you need to follow these basic steps:

1. Find the URL that you want to scrape
2. Inspecting the Page
3. Find the data you want to extract
4. Write the code
5. Run the code and extract the data
6. Store the data in the required format

Example: Scraping a website to get product details

Pre-requisite:

• Python 2.x or Python 3.x
• Selenium Library
• BeautifulSoup Library
• Pandas Library

1. We are going scrape online shopping website to extract the Price, Name, and rating of products, go to products URL
2. The data is usually nested in tags. So, we inspect the page to examine, under which tag the information we would like to scrape is nested. To inspect the page, just right click on the element and click on “Inspect”. When you click on the “Inspect” tab, you will see a “Browser Inspector Box” open.
3. Let’s extract the Price, Name, and Rating which is nested in the “div” tag respectively.
4. Write code:

#Let us import all the necessary libraries

from selenium import webdriver

from BeautifulSoup import BeautifulSoup

import pandas as pd

driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver")

products=[] #List to store name of the product

prices=[] #List to store price of the product

ratings=[] #List to store rating of the product

driver.get("Product_URL")

content = driver.page_source

soup = BeautifulSoup(content)

for a in soup.findAll('a',href=True, attrs={'class':'.…'}):

name=a.find('div', attrs={'class': '….'})

price=a.find('div', attrs={'class':'….'})

rating=a.find('div', attrs={'class':'….'})

products.append(name.text)

ratings.append(rating.text)

df = pd.DataFrame({'Product Name':products,'Price':prices,'Rating':ratings})

df.to_csv('products.csv', index=False, encoding='utf-8')

To run the code, a file name “products.csv” is created and this file contains the extracted data.