GeneralML/TextModel/Simple Text Mining concept and practice from scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple Text Mining concept and practice from scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook shows 3 different approches that could be taken when performing a text-mining, from it's concept and actual implementation of codes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "Text mining is an approach to find a relationship between two words in a given sentence. It could be found by using:\n",
    "1) Frequency of appearance of two words\n",
    "2) Statistical method of extracting connection\n",
    "3) Word2vec (DL)\n",
    "\n",
    "There are two prerequisite steps to do before performing text-mining\n",
    "1) Select the target word\n",
    "2) Choose the context: choose what is the sentence about\n",
    "\n",
    "Some of the visualization tools to display text mining results are\n",
    "1) Gephi\n",
    "2) Centriufuge\n",
    "3) Commetrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Frequency of appearance of two words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a method to count the number of the appearance of target words and other words in a given context. We asuume the higher the frequency, the higher the correlation. We usually set a threshold where we delete the pairs of words that does not reach that threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import glob\n",
    "from afinn import Afinn # Wordlist-based approach for sentiment analysis.\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "from nltk.tokenize import RegexpTokenizer\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Load the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data can be found in this kaggle [Link](https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews). It's a revised version of IMDB Dataset of 50K Movie Reviews. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = 'C:\\\\Users\\\\bokhy\\\\Desktop\\\\Python\\\\github\\\\Python-Projects'  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "error",
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'C:\\\\Users\\\\bokhy\\\\Desktop\\\\Python\\\\github\\\\Python-Projects\\\\IMDB Dataset.csv'",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-3-0e7fea6ff092>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mreview\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mos\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'IMDB Dataset.csv'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mengine\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m\"python\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mreview\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mhead\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pandas\\io\\parsers.py\u001b[0m in \u001b[0;36mparser_f\u001b[1;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)\u001b[0m\n\u001b[0;32m    674\u001b[0m         )\n\u001b[0;32m    675\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 676\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    677\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    678\u001b[0m     \u001b[0mparser_f\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pandas\\io\\parsers.py\u001b[0m in \u001b[0;36m_read\u001b[1;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[0;32m    446\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    447\u001b[0m     \u001b[1;31m# Create the parser.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 448\u001b[1;33m     \u001b[0mparser\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mTextFileReader\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfp_or_buf\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    449\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    450\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mchunksize\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0miterator\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pandas\\io\\parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, f, engine, **kwds)\u001b[0m\n\u001b[0;32m    878\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"has_index_names\"\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mkwds\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"has_index_names\"\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    879\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 880\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_make_engine\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mengine\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    881\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    882\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pandas\\io\\parsers.py\u001b[0m in \u001b[0;36m_make_engine\u001b[1;34m(self, engine)\u001b[0m\n\u001b[0;32m   1124\u001b[0m                     \u001b[1;34m'\"python-fwf\")'\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1125\u001b[0m                 )\n\u001b[1;32m-> 1126\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_engine\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mklass\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mf\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1127\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1128\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_failover_to_python\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pandas\\io\\parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, f, **kwds)\u001b[0m\n\u001b[0;32m   2267\u001b[0m             \u001b[0mencoding\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mencoding\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2268\u001b[0m             \u001b[0mcompression\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcompression\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 2269\u001b[1;33m             \u001b[0mmemory_map\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmemory_map\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   2270\u001b[0m         )\n\u001b[0;32m   2271\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mhandles\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mextend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mhandles\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pandas\\io\\common.py\u001b[0m in \u001b[0;36mget_handle\u001b[1;34m(path_or_buf, mode, encoding, compression, memory_map, is_text)\u001b[0m\n\u001b[0;32m    456\u001b[0m         \u001b[1;32melif\u001b[0m \u001b[0mis_text\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    457\u001b[0m             \u001b[1;31m# No explicit encoding\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 458\u001b[1;33m             \u001b[0mf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpath_or_buf\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m\"replace\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnewline\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m\"\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    459\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    460\u001b[0m             \u001b[1;31m# Binary mode\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'C:\\\\Users\\\\bokhy\\\\Desktop\\\\Python\\\\github\\\\Python-Projects\\\\IMDB Dataset.csv'"
     ]
    }
   ],
   "source": [
    "review = pd.read_csv(os.path.join(path, 'IMDB Dataset.csv'), engine=\"python\")\n",
    "review.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0     One of the other reviewers has mentioned that ...\n",
      "1     A wonderful little production. <br /><br />The...\n",
      "2     I thought this was a wonderful way to spend ti...\n",
      "3     Petter Mattei's \"Love in the Time of Money\" is...\n",
      "4     Probably my all-time favorite movie, a story o...\n",
      "                            ...                        \n",
      "95    I think this movie has got it all. It has real...\n",
      "96    Howard (Kevin Kline) teaches English at the hi...\n",
      "97    We usually think of the British as the experts...\n",
      "98    One of Starewicz's longest and strangest short...\n",
      "99    Nice character development in a pretty cool mi...\n",
      "Name: review, Length: 100, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# Only filter the positive reviews\n",
    "is_pos_review = review['sentiment'] == 'positive'\n",
    "pos_review = review[is_pos_review]['review'][0:100] # lets get 100 rows\n",
    "pos_review.reset_index(inplace=True, drop=True) # reset index\n",
    "print(pos_review)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = RegexpTokenizer('[\\w]+')\n",
    "stop_words = stopwords.words('english')\n",
    "\n",
    "count = {} # save the frequnecy of appearance as dict here\n",
    "for line in pos_review:\n",
    "    words = line.lower() # lower case\n",
    "    tokens = tokenizer.tokenize(words) # tokenize the words and save it\n",
    "    stopped_tokens = [i for i in list(set(tokens)) if not i in stop_words+[\"br\"]]\n",
    "    stopped_tokens2 = [i for i in stopped_tokens if len(i)>1]\n",
    "    for i,a in enumerate(stopped_tokens2): # index(i) and its corresponding value(a) in stopped_tokens2\n",
    "        for b in stopped_tokens2[i+1:]:\n",
    "            if a>b: # a and b are tokens. So it compares the the first character of words \n",
    "                count[b,a] = count.get((b,a),0) + 1\n",
    "            else:\n",
    "                count[a,b] = count.get((a,b),0) + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dictionary tip: **get**\n",
    "1) count.get((b,a)) : count&#50640;&#49436; key (b,a)&#50640; &#45824;&#51025;&#54616;&#45716; value&#47484; &#50619;&#45716;&#45796;. &#46357;&#49492;&#45320;&#47532;&#50640; (b,a)&#46972;&#45716; key&#44032; &#50630;&#51012; &#44221;&#50864; None&#51012; &#48152;&#54872;&#54620;&#45796;.\n",
    "2) count.get((b,a),0) : get&#51008; &#52572;&#45824; 2&#44060;&#51032; &#51064;&#49688;&#47484; &#48155;&#51012; &#49688; &#51080;&#45796;. &#46160; &#48264;&#51704; &#51064;&#49688;&#45716; &#44592;&#48376;&#44050;&#50640; &#54644;&#45817;&#54620;&#45796;. key &#47532;&#49828;&#53944;&#50640; (b,a)&#44032; &#51080;&#51012; &#44221;&#50864; value&#47484; &#48152;&#54872;&#54616;&#47728;, &#50630;&#51012; &#44221;&#50864; &#44592;&#48376;&#44050;&#50640; &#54644;&#45817;&#54616;&#45716; 0&#51012; &#48152;&#54872;&#54620;&#45796;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['one',\n",
       " 'of',\n",
       " 'the',\n",
       " 'other',\n",
       " 'reviewers',\n",
       " 'has',\n",
       " 'mentioned',\n",
       " 'that',\n",
       " 'after',\n",
       " 'watching']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.tokenize(pos_review[0].lower())[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### change dictionary to dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataframe become a single column dataframe, where index is a tuple corresponding to 'key' part in dictionary, and entry is a 'value' in dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>(mess, surreal)</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(may, mess)</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(become, mess)</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(get, mess)</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(mess, side)</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(slowly, taking)</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(pretty, slowly)</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(pound, taking)</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(pound, pretty)</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(pretty, taking)</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>694983 rows &times; 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  0\n",
       "(mess, surreal)   1\n",
       "(may, mess)       1\n",
       "(become, mess)    1\n",
       "(get, mess)       1\n",
       "(mess, side)      1\n",
       "...              ..\n",
       "(slowly, taking)  1\n",
       "(pretty, slowly)  1\n",
       "(pound, taking)   1\n",
       "(pound, pretty)   1\n",
       "(pretty, taking)  1\n",
       "\n",
       "[694983 rows x 1 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame.from_dict(count, orient='index')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we pair the two words that appear simultanously and its frequency in a new dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>term1</th>\n",
       "      <th>term2</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>film</td>\n",
       "      <td>one</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>like</td>\n",
       "      <td>movie</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>movie</td>\n",
       "      <td>one</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>film</td>\n",
       "      <td>like</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>film</td>\n",
       "      <td>story</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>movie</td>\n",
       "      <td>see</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>movie</td>\n",
       "      <td>time</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>one</td>\n",
       "      <td>time</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>like</td>\n",
       "      <td>one</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>movie</td>\n",
       "      <td>really</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>good</td>\n",
       "      <td>movie</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>film</td>\n",
       "      <td>see</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>film</td>\n",
       "      <td>way</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>one</td>\n",
       "      <td>see</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>film</td>\n",
       "      <td>movie</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>film</td>\n",
       "      <td>really</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>film</td>\n",
       "      <td>well</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>film</td>\n",
       "      <td>much</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>film</td>\n",
       "      <td>time</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>film</td>\n",
       "      <td>life</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    term1   term2  freq\n",
       "0    film     one    31\n",
       "1    like   movie    24\n",
       "2   movie     one    24\n",
       "3    film    like    23\n",
       "4    film   story    22\n",
       "5   movie     see    22\n",
       "6   movie    time    22\n",
       "7     one    time    21\n",
       "8    like     one    20\n",
       "9   movie  really    20\n",
       "10   good   movie    20\n",
       "11   film     see    20\n",
       "12   film     way    20\n",
       "13    one     see    20\n",
       "14   film   movie    20\n",
       "15   film  really    19\n",
       "16   film    well    19\n",
       "17   film    much    19\n",
       "18   film    time    19\n",
       "19   film    life    18"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list1=[]\n",
    "for i in range(len(df)):\n",
    "    list1.append([df.index[i][0], df.index[i][1], df[0][i]])\n",
    "\n",
    "df2 = pd.DataFrame(list1, columns=['term1','term2','freq'])\n",
    "df3 = df2.sort_values(by=['freq'], ascending=False) \n",
    "df3 = df3.reset_index(drop=True)\n",
    "df3.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### The same steps apply to the negative reviews as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "is_neg_review = review['sentiment'] == 'negative'\n",
    "neg_review = review[is_neg_review]['review'][0:100] # 100&#44060;&#47564; &#52628;&#52636;\n",
    "neg_review.reset_index(inplace=True, drop=True) # &#51064;&#45937;&#49828; &#52488;&#44592;&#54868;\n",
    "\n",
    "tokenizer = RegexpTokenizer('[\\w]+')\n",
    "stop_words = stopwords.words('english')\n",
    "\n",
    "count = {} \n",
    "for line in neg_review:\n",
    "    words = line.lower() \n",
    "    tokens = tokenizer.tokenize(words) \n",
    "    stopped_tokens = [i for i in list(set(tokens)) if not i in stop_words+[\"br\"]]\n",
    "    stopped_tokens2 = [i for i in stopped_tokens if len(i)>1]\n",
    "    for i,a in enumerate(stopped_tokens2):\n",
    "        for b in stopped_tokens2[i+1:]:\n",
    "            if a>b:\n",
    "                count[b,a] = count.get((b,a),0) + 1\n",
    "            else:\n",
    "                count[a,b] = count.get((a,b),0) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>term1</th>\n",
       "      <th>term2</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>like</td>\n",
       "      <td>movie</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>film</td>\n",
       "      <td>movie</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>movie</td>\n",
       "      <td>one</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>film</td>\n",
       "      <td>like</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>good</td>\n",
       "      <td>movie</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>like</td>\n",
       "      <td>one</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>film</td>\n",
       "      <td>one</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>good</td>\n",
       "      <td>like</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>even</td>\n",
       "      <td>movie</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>movie</td>\n",
       "      <td>would</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>movie</td>\n",
       "      <td>see</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>film</td>\n",
       "      <td>would</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>even</td>\n",
       "      <td>like</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>could</td>\n",
       "      <td>movie</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>film</td>\n",
       "      <td>see</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>movie</td>\n",
       "      <td>time</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>film</td>\n",
       "      <td>good</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>movie</td>\n",
       "      <td>movies</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>even</td>\n",
       "      <td>film</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>bad</td>\n",
       "      <td>movie</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    term1   term2  freq\n",
       "0    like   movie    47\n",
       "1    film   movie    37\n",
       "2   movie     one    34\n",
       "3    film    like    33\n",
       "4    good   movie    32\n",
       "5    like     one    31\n",
       "6    film     one    31\n",
       "7    good    like    29\n",
       "8    even   movie    29\n",
       "9   movie   would    28\n",
       "10  movie     see    27\n",
       "11   film   would    27\n",
       "12   even    like    27\n",
       "13  could   movie    26\n",
       "14   film     see    26\n",
       "15  movie    time    25\n",
       "16   film    good    25\n",
       "17  movie  movies    25\n",
       "18   even    film    24\n",
       "19    bad   movie    24"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame.from_dict(count, orient='index')\n",
    "\n",
    "list1=[]\n",
    "for i in range(len(df)):\n",
    "    list1.append([df.index[i][0], df.index[i][1], df[0][i]])\n",
    "\n",
    "df2 = pd.DataFrame(list1, columns=['term1','term2','freq'])\n",
    "df3 = df2.sort_values(by=['freq'], ascending=False) # freq &#44592;&#51456;&#51004;&#47196; &#45236;&#47548;&#52264;&#49692; &#51221;&#47148;\n",
    "df3 = df3.reset_index(drop=True)\n",
    "df3.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Statistical method of extracting connection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we would use cosine similarity.\n",
    "\n",
    "similarity is a quantitative statistical value, so we need weight the words to get the similarity. In simple words, we give adequate number value to each word. To do this, TF-IDF is often used \n",
    "\n",
    "$$\\mathrm{cosine\\ similarity}\\ S_{ij} = {A \\cdot B \\over ||A||\\ ||B||} = {\\sum_{k} x_{ik} \\times x_{jk} \\over \\sqrt{\\sum_{k} (x_{ik})^2} \\times \\sqrt{\\sum_{k} (x_{jk})^2}}$$$$\\mathrm{jaccard\\ similarity}\\ S_{ij} = {\\sum_k \\mathrm{min}(x_{ik}, x_{jk}) \\over \\sum_k \\mathrm{max}(x_{ik}, x_{jk})}$$$$\\mathrm{overlap\\ similarity}\\ S_{ij} = {\\sum_k \\mathrm{min}(x_{ik}, x_{jk}) \\over \\mathrm{min}(\\sum_k x_{ik}, \\sum_k x_{jk})}$$\n",
    "\n",
    "x: frequency\n",
    "i,j: word index\n",
    "k: sentence index "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import glob\n",
    "from afinn import Afinn\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "from nltk.tokenize import RegexpTokenizer\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from scipy import sparse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "review = pd.read_csv(os.path.join(path, 'IMDB Dataset.csv'), engine=\"python\")\n",
    "\n",
    "# get positive \n",
    "is_pos_review = review['sentiment'] == 'positive'\n",
    "pos_review = review[is_pos_review]['review'][0:100] \n",
    "pos_review.reset_index(inplace=True, drop=True) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<100x4995 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 10713 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stop_words = stopwords.words('english')\n",
    "vec = TfidfVectorizer(stop_words=stop_words)\n",
    "vector_pos_review = vec.fit_transform(pos_review)\n",
    "vector_pos_review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>4985</th>\n",
       "      <th>4986</th>\n",
       "      <th>4987</th>\n",
       "      <th>4988</th>\n",
       "      <th>4989</th>\n",
       "      <th>4990</th>\n",
       "      <th>4991</th>\n",
       "      <th>4992</th>\n",
       "      <th>4993</th>\n",
       "      <th>4994</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.146447</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.085182</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.072555</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows &times; 4995 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    0     1     2     3     4     5         6     7     8     9     ...  4985  \\\n",
       "0    0.0   0.0   0.0   0.0   0.0   0.0  0.000000   0.0   0.0   0.0  ...   0.0   \n",
       "1    0.0   0.0   0.0   0.0   0.0   0.0  0.000000   0.0   0.0   0.0  ...   0.0   \n",
       "2    0.0   0.0   0.0   0.0   0.0   0.0  0.000000   0.0   0.0   0.0  ...   0.0   \n",
       "3    0.0   0.0   0.0   0.0   0.0   0.0  0.000000   0.0   0.0   0.0  ...   0.0   \n",
       "4    0.0   0.0   0.0   0.0   0.0   0.0  0.146447   0.0   0.0   0.0  ...   0.0   \n",
       "..   ...   ...   ...   ...   ...   ...       ...   ...   ...   ...  ...   ...   \n",
       "95   0.0   0.0   0.0   0.0   0.0   0.0  0.000000   0.0   0.0   0.0  ...   0.0   \n",
       "96   0.0   0.0   0.0   0.0   0.0   0.0  0.000000   0.0   0.0   0.0  ...   0.0   \n",
       "97   0.0   0.0   0.0   0.0   0.0   0.0  0.000000   0.0   0.0   0.0  ...   0.0   \n",
       "98   0.0   0.0   0.0   0.0   0.0   0.0  0.000000   0.0   0.0   0.0  ...   0.0   \n",
       "99   0.0   0.0   0.0   0.0   0.0   0.0  0.000000   0.0   0.0   0.0  ...   0.0   \n",
       "\n",
       "    4986  4987  4988  4989  4990  4991      4992      4993  4994  \n",
       "0    0.0   0.0   0.0   0.0   0.0   0.0  0.000000  0.000000   0.0  \n",
       "1    0.0   0.0   0.0   0.0   0.0   0.0  0.000000  0.000000   0.0  \n",
       "2    0.0   0.0   0.0   0.0   0.0   0.0  0.000000  0.000000   0.0  \n",
       "3    0.0   0.0   0.0   0.0   0.0   0.0  0.000000  0.000000   0.0  \n",
       "4    0.0   0.0   0.0   0.0   0.0   0.0  0.000000  0.000000   0.0  \n",
       "..   ...   ...   ...   ...   ...   ...       ...       ...   ...  \n",
       "95   0.0   0.0   0.0   0.0   0.0   0.0  0.000000  0.000000   0.0  \n",
       "96   0.0   0.0   0.0   0.0   0.0   0.0  0.000000  0.000000   0.0  \n",
       "97   0.0   0.0   0.0   0.0   0.0   0.0  0.000000  0.085182   0.0  \n",
       "98   0.0   0.0   0.0   0.0   0.0   0.0  0.072555  0.000000   0.0  \n",
       "99   0.0   0.0   0.0   0.0   0.0   0.0  0.000000  0.000000   0.0  \n",
       "\n",
       "[100 rows x 4995 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# change sparse matrix to regular dataframe\n",
    "A = vector_pos_review.toarray()\n",
    "pd.DataFrame(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>90</th>\n",
       "      <th>91</th>\n",
       "      <th>92</th>\n",
       "      <th>93</th>\n",
       "      <th>94</th>\n",
       "      <th>95</th>\n",
       "      <th>96</th>\n",
       "      <th>97</th>\n",
       "      <th>98</th>\n",
       "      <th>99</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.074655</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.074307</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4990</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4991</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4992</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.072555</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4993</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.085182</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4994</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4995 rows &times; 100 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       0    1    2    3    4         5    6    7    8    9   ...   90   91  \\\n",
       "0     0.0  0.0  0.0  0.0  0.0  0.000000  0.0  0.0  0.0  0.0  ...  0.0  0.0   \n",
       "1     0.0  0.0  0.0  0.0  0.0  0.000000  0.0  0.0  0.0  0.0  ...  0.0  0.0   \n",
       "2     0.0  0.0  0.0  0.0  0.0  0.000000  0.0  0.0  0.0  0.0  ...  0.0  0.0   \n",
       "3     0.0  0.0  0.0  0.0  0.0  0.074655  0.0  0.0  0.0  0.0  ...  0.0  0.0   \n",
       "4     0.0  0.0  0.0  0.0  0.0  0.000000  0.0  0.0  0.0  0.0  ...  0.0  0.0   \n",
       "...   ...  ...  ...  ...  ...       ...  ...  ...  ...  ...  ...  ...  ...   \n",
       "4990  0.0  0.0  0.0  0.0  0.0  0.000000  0.0  0.0  0.0  0.0  ...  0.0  0.0   \n",
       "4991  0.0  0.0  0.0  0.0  0.0  0.000000  0.0  0.0  0.0  0.0  ...  0.0  0.0   \n",
       "4992  0.0  0.0  0.0  0.0  0.0  0.000000  0.0  0.0  0.0  0.0  ...  0.0  0.0   \n",
       "4993  0.0  0.0  0.0  0.0  0.0  0.000000  0.0  0.0  0.0  0.0  ...  0.0  0.0   \n",
       "4994  0.0  0.0  0.0  0.0  0.0  0.000000  0.0  0.0  0.0  0.0  ...  0.0  0.0   \n",
       "\n",
       "            92   93   94   95   96        97        98   99  \n",
       "0     0.000000  0.0  0.0  0.0  0.0  0.000000  0.000000  0.0  \n",
       "1     0.000000  0.0  0.0  0.0  0.0  0.000000  0.000000  0.0  \n",
       "2     0.000000  0.0  0.0  0.0  0.0  0.000000  0.000000  0.0  \n",
       "3     0.074307  0.0  0.0  0.0  0.0  0.000000  0.000000  0.0  \n",
       "4     0.000000  0.0  0.0  0.0  0.0  0.000000  0.000000  0.0  \n",
       "...        ...  ...  ...  ...  ...       ...       ...  ...  \n",
       "4990  0.000000  0.0  0.0  0.0  0.0  0.000000  0.000000  0.0  \n",
       "4991  0.000000  0.0  0.0  0.0  0.0  0.000000  0.000000  0.0  \n",
       "4992  0.000000  0.0  0.0  0.0  0.0  0.000000  0.072555  0.0  \n",
       "4993  0.000000  0.0  0.0  0.0  0.0  0.085182  0.000000  0.0  \n",
       "4994  0.000000  0.0  0.0  0.0  0.0  0.000000  0.000000  0.0  \n",
       "\n",
       "[4995 rows x 100 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we use tranpose to change it from sentence-sentence similarity to word-sentence similarity\n",
    "A=A.transpose()\n",
    "pd.DataFrame(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[((1098, 133), 0.3484457223054042),\n",
       " ((1099, 133), 0.4726439154636967),\n",
       " ((1104, 133), 0.3382677891921974),\n",
       " ((1133, 133), 0.2738023270540216),\n",
       " ((1134, 133), 0.17092631488618495),\n",
       " ((1154, 133), 0.4726439154636967),\n",
       " ((1173, 133), 0.4726439154636967),\n",
       " ((1184, 133), 0.4726439154636967),\n",
       " ((1209, 133), 0.1523951677290328),\n",
       " ((1211, 133), 0.8812534988158323)]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Change it again to Spare matrix\n",
    "A_sparse = sparse.csr_matrix(A)\n",
    "similarities_sparse = cosine_similarity(A_sparse, dense_output=False)\n",
    "list(similarities_sparse.todok().items())[35000:35010]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dead\n",
      "affected\n"
     ]
    }
   ],
   "source": [
    "print(vec.get_feature_names()[1098])\n",
    "print(vec.get_feature_names()[133])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['active', 'activities', 'actor', 'actors', 'actress']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec.get_feature_names()[100:105]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>words</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(616, 1511)</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(1511, 616)</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(2929, 2082)</td>\n",
       "      <td>0.499995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(2082, 2929)</td>\n",
       "      <td>0.499995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(3483, 69)</td>\n",
       "      <td>0.499987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>(4701, 3483)</td>\n",
       "      <td>0.499987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>(3483, 1886)</td>\n",
       "      <td>0.499987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>(2033, 3483)</td>\n",
       "      <td>0.499987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>(4680, 3483)</td>\n",
       "      <td>0.499987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>(3483, 4987)</td>\n",
       "      <td>0.499987</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          words    weight\n",
       "0   (616, 1511)  0.500000\n",
       "1   (1511, 616)  0.500000\n",
       "2  (2929, 2082)  0.499995\n",
       "3  (2082, 2929)  0.499995\n",
       "4    (3483, 69)  0.499987\n",
       "5  (4701, 3483)  0.499987\n",
       "6  (3483, 1886)  0.499987\n",
       "7  (2033, 3483)  0.499987\n",
       "8  (4680, 3483)  0.499987\n",
       "9  (3483, 4987)  0.499987"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get dataframe in desc order in similarity\n",
    "df = pd.DataFrame(list(similarities_sparse.todok().items()), columns=['words', 'weight'])\n",
    "df2 = df.sort_values(by=['weight'], ascending=False)\n",
    "df2 = df2.reset_index(drop=True)\n",
    "df3 = df2.loc[np.round(df2['weight']) < 1]\n",
    "df3 = df3.reset_index(drop=True)\n",
    "\n",
    "df3.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Word2vec (DL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import glob\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "from nltk.tokenize import RegexpTokenizer\n",
    "import numpy as np\n",
    "from gensim.models.word2vec import Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First few steps are similar to what we did in another approach"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "review = pd.read_csv(os.path.join(path, 'IMDB Dataset.csv'), engine=\"python\")\n",
    "\n",
    "# get positive \n",
    "is_pos_review = review['sentiment'] == 'positive'\n",
    "pos_review = review[is_pos_review]['review'][0:100] \n",
    "pos_review.reset_index(inplace=True, drop=True) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = RegexpTokenizer('[\\w]+')\n",
    "stop_words = stopwords.words('english')\n",
    "\n",
    "text = [] \n",
    "for line in pos_review:\n",
    "    words = line.lower() \n",
    "    tokens = tokenizer.tokenize(words) \n",
    "    stopped_tokens = [i for i in list(set(tokens)) if not i in stop_words+[\"br\"]]\n",
    "    stopped_tokens2 = [i for i in stopped_tokens if len(i)>1]\n",
    "    text.append(stopped_tokens2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Word2Vec(text, \n",
    "                 sg=1, #enable skip-gram\n",
    "                 window=2, # apply it to both left right side at max of 2 words\n",
    "                 min_count=3) # onyl use words that appeared more than 3 times\n",
    "model.init_sims(replace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.70896316"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This shows the similarity between 'film' and 'movie'\n",
    "model.wv.similarity('film', 'movie')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('time', 0.7345631122589111),\n",
       " ('work', 0.7224764823913574),\n",
       " ('well', 0.7197010517120361),\n",
       " ('movie', 0.7130800485610962),\n",
       " ('film', 0.6941635012626648)]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Top 5 words that are most close to the word \"good\" \n",
    "model.wv.most_similar(\"good\", topn=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}