Task2 数据读取与数据分析.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在上一章节，我们给大家简单介绍了赛题的内容和几种解决方案。从本章开始我们将会逐渐带着大家使用思路1到思路4来完成本次赛题。在讲解工具使用的同时，我们还会讲解一些算法的原理和相关知识点，并会给出一定的参考文献供大家深入学习。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Task2 数据读取与数据分析**\n",
    "\n",
    "本章主要内容为数据读取和数据分析，具体使用`Pandas`库完成数据读取操作，并对赛题数据进行分析构成。\n",
    "\n",
    "### **学习目标**\n",
    "\n",
    "- 学习使用`Pandas`读取赛题数据\n",
    "- 分析赛题数据的分布规律\n",
    "\n",
    "### **数据读取**\n",
    "\n",
    "赛题数据虽然是文本数据，每个新闻是不定长的，但任然使用csv格式进行存储。因此可以直接用`Pandas`完成数据读取的操作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200001 data/train_set.csv\r\n"
     ]
    }
   ],
   "source": [
    "!wc -l data/train_set.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2967 6758 339 2021 1854 3731 4109 3792 4149 15...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>4464 486 6352 5619 2465 4802 1452 3137 5778 54...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>7346 4068 5074 3747 5681 6093 1777 2226 7354 6...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>7159 948 4866 2109 5520 2490 211 3956 5520 549...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>3646 3055 3055 2490 4659 6065 3370 5814 2465 5...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   label                                               text\n",
       "0      2  2967 6758 339 2021 1854 3731 4109 3792 4149 15...\n",
       "1     11  4464 486 6352 5619 2465 4802 1452 3137 5778 54...\n",
       "2      3  7346 4068 5074 3747 5681 6093 1777 2226 7354 6...\n",
       "3      2  7159 948 4866 2109 5520 2490 211 3956 5520 549...\n",
       "4      3  3646 3055 3055 2490 4659 6065 3370 5814 2465 5..."
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "train_df = pd.read_csv('data/train_set.csv', sep='\\t',nrows=10**4)\n",
    "print(train_df.shape)\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "label     int64\n",
       "text     object\n",
       "dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里的`read_csv`由三部分构成：\n",
    "\n",
    "- 读取的文件路径，这里需要根据改成你本地的路径，可以使用相对路径或绝对路径；\n",
    "\n",
    "- 分隔符`sep`，为每列分割的字符，设置为`\\t`即可；\n",
    "- 读取行数`nrows`，为此次读取文件的函数，是数值类型（由于数据集比较大，建议先设置为100）；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图是读取好的数据，是表格的形式。第一列为新闻的类别，第二列为新闻的字符。\n",
    "\n",
    "### **数据分析**\n",
    "\n",
    "在读取完成数据集后，我们还可以对数据集进行数据分析的操作。虽然对于非结构数据并不需要做很多的数据分析，但通过数据分析还是可以找出一些规律的。\n",
    "\n",
    "\n",
    "\n",
    "此步骤我们读取了所有的训练集数据，在此我们通过数据分析希望得出以下结论：\n",
    "\n",
    "- 赛题数据中，新闻文本的长度是多少？\n",
    "- 赛题数据的类别分布是怎么样的，哪些类别比较多？\n",
    "- 赛题数据中，字符分布是怎么样的？\n",
    "\n",
    "\n",
    "\n",
    "#### **句子长度分析**\n",
    "\n",
    "在赛题数据中每行句子的字符使用空格进行隔开，所以可以直接统计单词的个数来得到每个句子的长度。统计并如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count    10000.000000\n",
      "mean       908.766300\n",
      "std       1033.708515\n",
      "min         15.000000\n",
      "25%        375.000000\n",
      "50%        672.500000\n",
      "75%       1123.000000\n",
      "max      44665.000000\n",
      "Name: text_len, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# %pylab inline\n",
    "train_df['text_len'] = train_df['text'].apply(lambda x: len(x.split(' ')))\n",
    "print(train_df['text_len'].describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f5d6e3b1518>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD2CAYAAADbPoDqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAdc0lEQVR4nO3de5Cc1Znf8e/Tl7nqihgGhJAFaxDYgACrWGQUPBDLNsnaSylOWbETxeW1SVJ4K44rWaCsTQhog5fa3cK7a+xoV7tx2WUXWpfBi72AyoBAxCJeiZvBQCGzAksga9BtrJFmprvfJ3909/So1TN9ZqZnRtPn9ylU9Bydfvu8r3r66efcXnN3REQkXqmZboCIiMwsBQIRkcgpEIiIRE6BQEQkcgoEIiKRy8x0A8brzDPP9GXLls10M0REZpVdu3a96+5dtf5u1gWCZcuWsXPnzpluhojIrGJmb472d+oaEhGJnAKBiEjkFAhERCI368YIRCQ+uVyOvXv3MjAwMNNNOe21tbWxZMkSstls8HMUCETktLd3717mzp3LsmXLMLOZbs5py905ePAge/fu5fzzzw9+nrqGROS0NzAwwKJFixQE6jAzFi1aNO7MSYFARGYFBYEwE7lOCgQiIpFTIACOnsjx2//rJzz71uGZboqIyLRTIAB+3TfAr/sG2X3g2Ew3RUROU88//zzPP//8lD6vp6dn3MdvBM0aAgZyBQByhWSGWyIi9fzPh17mF2/3NfSY71s8j//x8fePWaf8YX7FFVeM69gTfd50UiAABvPFADCUVyAQkVPdfvvtPPDAAwB8+9vf5qGHHmL9+vUcOHCAyy67jK9//es8/fTTfO1rX+P+++/n2muvZcuWLdx3330nPe+xxx4Lfs3jx4+f8hp33HEHuVyO7du309fXxyOPPMLZZ5896fNTIAAGcwoEIrNFvW/uU+Huu+9m+fLlAHz2s5/l3nvv5dJLL+WOO+5g7dq1vPjii6xevZrNmzfzxS9+kZtuuonzzjvvlOeNx6ZNm055DYDdu3fz1FNPceedd/L444/z6U9/etLnp0AADObVNSQi4V577TV++tOfsm3bNo4cOcK+ffu4/PLLueWWW1i1ahW9vb1T8hoA69evB2Dp0qUMDQ1N+nVAg8UADCgjEJE62tvbOX78OAAXXXQRX/rSl9i2bRsbN25k6dKlAGzcuJHbb7+du+66q+bz3D349ZYvX17zNTo7Oxt1SsMUCKhkBEOF8H8kEYnLmjVr+MEPfsC1117LVVddxcMPP8x1113HN7/5Tc477zy+//3vs3jxYu68805efvllnn322VOet3379uDX+8IXvnDKa0wVG0+EOh2sXLnSG31jmu/97C1u/8HP+dy15/PfP/6+hh5bRCbvlVde4ZJLLpnpZswata6Xme1y95W16muMAE0fFZHpU71WYP78+fzwhz+cmcaUKBCg6aMis4G7N8V+Q9u2bZvS40+kl0djBFSmjyojEDk9tbW1cfDgwQl9yMWkvA11W1vbuJ6njIDKYPGgAoHIaWnJkiXs3bu3IdMym135xjTjoUBAZfpoTl1DIqelbDY7rhutyPioa4iR00cVCEQkPgoEVAaLNUYgIjFSIECzhkQkbgoEVNYRaGWxiMRIgQBlBCISt6BAYGabzWyHmW0YT53QslL5fWb28YmcxGQNamWxiESsbiAws7VA2t1XAReY2YUhdULLSs//Z8DZ7v5QA88t2IAyAhGJWEhG0ANsKT3eCqwOrBNUZmZZ4K+APWb2u7UaYGY3m9lOM9s5FQtKlBGISMxCAkEnsK/0+BDQHVgntGw98AvgHuBqM/v96oO7+yZ3X+nuK7u6ugKaPD5DyghEJGIhgeAY0F56PGeU59SqE1p2JbDJ3fcD3wGuH98pTN7wYLEyAhGJUEgg2EWlO2gFsCewTmjZbuCCUtlK4M2wpjfO8PRRZQQiEqGQvYYeBLab2WLgRmCdmW109w1j1LkG8MCyBPgbM1sHZIFPNubUwmllsYjErG4gcPc+M+sB1gD3lLpwXqhT5yhAaBnwrxtxMhNV3msoccgXEjJpLa8QkXgE7T7q7oepzPYJrhNaNpMKiZMrOO3ZNCdyBXIFJ5Oe6VaJiEyf6L/6lrOBuW3FmKhxAhGJjQJB6V4Ew4FA4wQiEhkFgnw5EGQBBQIRiU/0gaA8dbScEeguZSISm+gDQSUjUNeQiMRJgaA8WNxa6hpSRiAikVEgUEYgIpGLPhCUxwjmaIxARCIVfSCoTB/VrCERiZMCQVXXkPYbEpHYKBCUBovnaWWxiEQq+kAwUOoamlOeNVTwmWyOiMi0iz4QaK8hEYmdAoHGCEQkctEHgurpo8oIRCQ20QeCwXxCNm20ZYs3IVBGICKxUSDIJbRl0rSU7ko2qIxARCKjQJAv0JpNDQcCZQQiEpvoA8FALqE1kyaVMjIp0xiBiEQn+kAwmC/QmilehpZMShmBiERHgSCf0FoaKM6mU8oIRCQ60QeCgdzJGYFWFotIbIICgZltNrMdZrZhPHVCyswsY2Zvmdm20p/LJnNC4zWYTyqBQBmBiESobiAws7VA2t1XAReY2YUhdULLgMuB77l7T+nPzxt5gvUM5pPhNQQaIxCRGIVkBD3AltLjrcDqwDqhZdcAv2NmPytlC5nqg5vZzWa208x29vb2BjQ53OCIrqFsWrOGRCQ+IYGgE9hXenwI6A6sE1r2j8CH3f1qIAv8i+qDu/smd1/p7iu7uroCmhxu5GCxMgIRidEp375rOAa0lx7PoXbwqFUntOxFdx8sle0ETul6mkonZwQp3aFMRKITkhHsotIdtALYE1gntOzbZrbCzNLATcAL42j/pBXHCDRYLCLxCskIHgS2m9li4EZgnZltdPcNY9S5BvDAsheB7wIG/L27/6QxpxamOGuo0jV0bDA/nS8vIjLj6gYCd+8zsx5gDXCPu++n6lt7jTpHAQLLjlKcOTQjTlpHkNYYgYjEJyQjwN0PU5ntE1wntGym5AsJ+cSHMwKtLBaRGEW9srg8MDw8RpBJkdPKYhGJTNSBoHzj+pNmDSkjEJHIRB0IyjeuH7mOQNNHRSQ2cQeCXFXXkFYWi0iE4g4E+XLXkFYWi0i8og4EA7lS15DGCEQkYlEHgloZQT5xkkQzh0QkHpEHgmJGUB4jyJZuYK8BYxGJSdyBIHdyRlDuItI4gYjEJOpAMDA8fbQqI9A4gYhEJOpAMFi1oKxlOCPQGIGIxCPuQJAvryOo7DUEyghEJC5RB4Lq6aPljECDxSISk6gDwSnTR9MGKCMQkbhEHghqZwSaNSQiMYk8ECRk00YqVcwEtI5ARGIUdSAYyie0pCuXoPw4p64hEYlI1IGgkDjpUjYAkC11DQ0qIxCRiEQdCHKFZLg7CJQRiEicog4EhcTJpCsZgaaPikiMog4EuYKTSdXICBQIRCQiUQeCfJKclBGUxwi0jkBEYhIUCMxss5ntMLMN46kTWlYq7zaz5yZyEhOVT5zMiMHiFm0xISIRqhsIzGwtkHb3VcAFZnZhSJ3QshGH+ROgvREnFSpfSGp2DQ1p0zkRiUhIRtADbCk93gqsDqwTWoaZ3QD0A/trNcDMbjaznWa2s7e3N6DJYfKFUQaLlRGISERCAkEnsK/0+BDQHVgnqMzMWoA/BG4brQHuvsndV7r7yq6uroAmh6nuGsqWgoIGi0UkJiGB4BiVLps5ozynVp3QstuA+9z9yHgbP1nFweLK6WTSKVKmjEBE4hISCHZR6Q5aAewJrBNa9mHgFjPbBlxhZn8d3vzJyRdOzgiguN+QMgIRiUkmoM6DwHYzWwzcCKwzs43uvmGMOtcAHlLm7t8tH8TMtrn75xtxYiHyiQ/fuL6sJZMa3p5aRCQGdTMCd++jOMj7DHC9u79QFQRq1TkaWlZ1nJ5Jns+4VM8aguLMIWUEIhKTkIwAdz9MZbZPcJ3QsplSPVgMxYxAYwQiEpO4VxZXTR8FjRGISHziDgRVs4aglBEoEIhIRCIPBLVnDQ3ltbJYROIRdyCo2n0UlBGISHziDgRJcupgcdoYKt3UXkQkBnEHghqDxW3ZNAM5ZQQiEo+oA0H1rSoBOlsyHB/Kz1CLRESmX9SBoPrm9QAdrWn6B9U1JCLxiDoQ5JJTu4aUEYhIbKIOBIXEyVbNGupoTdM/pIxAROIRbSBw95pdQ3NaMgzlE60uFpFoRBsIcqXbUWbT1WMExe2XjisrEJFIRBsICkkxEKRT1bOG0gD0D2qcQETiEG0gyCXFrp/RMwIFAhGJQ7SBoFDqGqpeWVzJCNQ1JCJxiDYQlDOCdNWCso6WYkbQr4xARCIRbSDIlweLqzOC1mJGcFwZgYhEItpAUBksrg4EyghEJC7RBoLyOoFaew2Bpo+KSDyiDQT5UkZQvcVER6umj4pIXOINBKPMGurIataQiMTltAgEZnaGma0xszOn6zXzpVlD1Xcoy6RTtGZSWkcgItEICgRmttnMdpjZhvHUCSkzs4XAj4CrgSfMrGvCZzMOo3UNQXHAWIPFIhKLuoHAzNYCaXdfBVxgZheG1AktAy4HvuzufwQ8ClzVuNMbXaVr6NRL0NGS1vRREYlGSEbQA2wpPd4KrA6sE1Tm7k+6+zNmdh3FrGBH9cHN7GYz22lmO3t7ewOaXF++NGuoZkbQooxAROIREgg6gX2lx4eA7sA6oWWYmQGfAg4DueqDu/smd1/p7iu7uhrTczTcNZSq1TWU1vRREYlGSCA4BrSXHs8Z5Tm16oSW4UW3AC8CnxjfKUzM8GBx+tTT6WzNaPqoiEQjJBDsotIdtALYE1gnqMzMbjWz9aWyBcCR4NZPQm6U6aNQGiNQRiAikcgE1HkQ2G5mi4EbgXVmttHdN4xR5xrAA8tSwBYz+zzwEsWxgylXGGvWkMYIRCQidQOBu/eZWQ+wBrjH3fcDL9SpcxQgtKz087QqbzFRc9ZQa1oLykQkGiEZAe5+mMpsn+A6oWUzoTDWYHGLxghEJB6nxcrimTC8jqBG11BHS4bBfDI8xVREpJlFGwgqt6qsNWuodE+CnLqHRKT5RRsIRrsfAVTuSaDVxSISg2gDQW74DmW1t5gA3ZxGROIQbSAoDN+zuPZgMSgjEJE4RBsIxlxQ1qqMQETiEW0gGL55fa3B4lJGoCmkIhKDaANBuWuoRkIwPGuoX9tMiEgEog0EucTJpo3ixqcn6xgeI1BGICLNL9pAkC8kNbeXgMr0UWUEIhKDeANB4jUHiqEyfVQZgYjEIN5AUPCa20tAcQC5JZNSRiAiUYg3ECROepSuIYDOljTHNX1URCIQbyAoJGRHyQigOGCsrahFJAbxBoJk9K4hKE4h1ToCEYlB3IFgjK6hDt2lTEQiEW8gKCSjzhqCYkag+xaLSAziDQSJ19yCukx3KRORWMQbCApJzX2GyjpbM8oIRCQK8QaCOoPFHZo+KiKRiDcQFEZfWQzFjEDTR0UkBvEGgmT0vYagmBGcyBWGb2kpItKsggKBmW02sx1mtmE8dULKzGy+mT1sZlvN7AEza5nMCYXKjbHFBIy4S5m6h0SkydUNBGa2Fki7+yrgAjO7MKROaBnwGeDP3P0jwH7gY407vdEVxth0Dip3KdOAsYg0u0xAnR5gS+nxVmA18HpAnStDytz9vhHH6QIOBLd+EnKFhMwYs4bmlLaiPjaYp3s6GiQiMkNCuoY6gX2lx4eg5udirTqhZQCY2Spgobs/U31wM7vZzHaa2c7e3t6AJtdXLyOY154F4OiJXENeT0TkdBUSCI4B7aXHc0Z5Tq06oWWY2RnAXwCfq9UAd9/k7ivdfWVXV1dAk+srTh8d/fQXdhSHKo4cH2rI64mInK5CAsEuit06ACuAPYF1gspKg8N/B9zu7m+Oq/WTkCskZMfICBaUMoIjx5URiEhzCxkjeBDYbmaLgRuBdWa20d03jFHnGsADy34PuAr4ipl9BfiGu9/fmNMbXaHOFhPljOCwAoGINLm6gcDd+8ysB1gD3OPu+4EX6tQ5ChBY9o3Sn2lVnD46ekI0ty1DyuCouoZEpMmFZAS4+2Eqs32C64SWzYRCMvbuo6mUMb89q4xARJpevCuL6ywoA1jQ0cIRzRoSkSYXbSDIJWPvPgowvz2rWUMi0vSiDQT1BosBFnZkNWtIRJpelIHA3ckVfMzpo1DuGlJGICLNLcpAUN5RND3G7qMACzqyHOlXRiAizS3KQJAvBYK6g8XtLfxmME+ukExHs0REZkTUgSBbJxAs7NR+QyLS/KIMBIVCWNfQfG0zISIRiDIQ5JJiV0/djEAbz4lIBKIMBPlSRjDWrSqhOFgMyghEpLnFGQhKGcFYW0xAcbAY4LAyAhFpYnEGgkLgrCENFotIBOIMBKWMoN7K4rmtGdIpU9eQiDS1SANBefro2KdvZixoz6prSESaWpyBYHiweOyMAGB+R1Y7kIpIU4szEASuLIbiFFJNHxWRZhZnICiUZw3VP/0F7dqBVESaW5SBIDeOrqEFHS0KBCLS1KIMBIXhrqGAjKBDN6cRkeYWZSAobzERMkawoD1L/1CBobx2IBWR5hRlICiMp2uos7TfkG5QIyJNKspAUNliImywGOCoxglEpElFGQjKg8X1dh+Fyg6khxUIRKRJBQUCM9tsZjvMbMN46oyjrNvMtk/0JMarcqvKkFlD5R1I1TUkIs2pbiAws7VA2t1XAReY2YUhdcZRthD4FtDZyBMbS/nWk/W2mABtRS0izS8kI+gBtpQebwVWB9YJLSsAnwL6RmuAmd1sZjvNbGdvb29Ak8eWH1dGoMFiEWluIYGgE9hXenwI6A6sE1Tm7n3ufnSsBrj7Jndf6e4ru7q6Apo8tvFsMdHZkiabNg4eUyAQkeYUEgiOAe2lx3NGeU6tOqFl0668xUQ2YNaQmXHpufP5v798d6qbJSIyI0I+iHdR6Q5aAewJrBNaNu2GB4sDMgKAj77/bF7a18few8enslkiIjMiE1DnQWC7mS0GbgTWmdlGd98wRp1rAA8sm3bD00cDMgIoBoKvPvwqW1/+NZ9bff5UNk1EZNrV/SR09z6Kg7zPANe7+wtVQaBWnaOhZSOO0dOA8wlS7hoKGSwGOP/MTi7qnsOjL++fymaJiMyIoK/E7n7Y3be4+6ifhLXqhJZNt8odysICARSzgn/cc4iDxwanqlkiIjMiypXF+SQhnTLMxhcIEofHXjkwhS0TEZl+kQYCD+4WKnv/4nmcu6Bd3UMi0nTiDAQFJzvOQGBmrHlfN9t3v8uJocIUtUxEZPpFGgiScWcEAP/8krMYyifseENrCkSkecQZCBIP2meo2tXnn0F7Ns0Tr05+mwsRkdNFnIGg4EHbS1RrzaS59r2LeOK1A7j7FLRMRGT6RRkIckkSdFOaWnqWn8Xewyf4ZW9/g1slIjIzogwEhWRiGQFAz/LipnfbXtM0UhFpDlEGgnzBg+5XXMuShR1c1D2HJxQIRKRJxBkIJtE1BMXuoZ/90yH6B/MNbJWIyMyIMxBMcLC4rGd5F7mC88hLWlwmIrNflIEgl0y8awjgt89fxKXnzuOPH3mV3wzoFpYiMrtFGQgKSUJmAusIytIp467fvZTeY4Pc+5PXG9gyEZHpF2UgyE1isLjsyqUL+TdXL+X//HQPr7wz6u2WRUROe1EGgslMHx3pDz66nPntWTY8+BJJogVmIjI7RRUIyjekyRcmN2uobEFHC7fdeDG73jzM95/dO+njiYjMhGgCwaMv7+cDG3/Cgb6BhnQNlX3yqiWsfM9C7v6HVzjcP9SQY4qITKdoAsFvdXVy9ESOH//8nYZ1DQGkUsZdN11K30Ceex59rSHHFBGZTtEEgveeNZdLzpnH37/wdnGvoUnMGqp2yTnz+OwHl/G9n73Ff/rOLnYf+E3Dji0iMtUyM92A6fSJFYv540depbMlzWXnzm/osf/bR5fT2Zph8/Y3ePTl/dxwcTef/MC5XH/xWbRm0g19LRGRRoomIwD4ncvPAaB/qNCQweKR2rJpvrzmIp76g+v5Dx/6LV7ce4T/+J1nWXX34/zp1tc40DfQ0NcTkeb39pETfPPJXzKQm9q7IkaVEZx3RgdXLl3Ac28dIdugMYJqi+a0cuvHLua/fmQ5T+9+l+888yZ/+cRuvrHtl1y1dCEffO8iVixZwJKF7Zw9v43OlgypBg1ci8js5e7sO3KCc+a3k04Z7x4b5DN//f/4p3f7efbNw9z3masa2qU9UlAgMLPNwPuAH7v7xtA6kymbKp9YsZjn3joyoVtVjkc6ZXzooi4+dFEXe97t5/6dv+Lp19/la4+9TvU9bdqyKbrntbH0jA7OmttGSyZFS9qK/8+kaEmnyWaMlnSK1lJZ19xWlizsoHtuG63ZFC3pFIP5hONDeRxKzyv+SaUMdydfWutQnjF1qH+It48MkEkb5y5sZ15bdsxzcncG8wkDuQIncgXyBactm6a9JU1bJjXpN2khcY4N5nF3MukUmZSRSRnplGHW/MHS3Um8+H8H3MHx4fdLTNdiogqJkzKGr5G7kys42XTluuULCfnEac2kMCv+bgzkEhynPZvGzMgXEn4zkCedNua0ZDAr9iQcPZGjsyXNvLYsiTsH+4c4eiLHwo4WFnW2cGwoz1sHj9M/mOe8MzrontfGmwf7eent4qLTSxfPo2tuK9tff5cnX+ula24rH3l/N/nEueeRV3nmjUNceNYc/vOHL+R/P/kG7xw9wb9f9R6+teNNvvLAS3z1X102Jf/+dQOBma0F0u6+ysz+xswudPfX69UBLptoWfXxG+lfXnYOd/3oF7Rkpq9XbNmZndz6sYu59WNw9HiO3b3H2Hv4OL/uG6B/sMDxoTzvHB3gV4eO80ZvP4P5hKF8gVzBGSokFCa5WC2dMhKvfKCkrFiWK5x83LZsivQob7LEYSBfOCWIjZRNG9l0imTEB1q56eWjmoFhlP4b/tkp/jKOJpMyMmkjVaN9tdpU/CitUT7OS1nrchinFo72u1n+ME8cGPHBXvygr3zgj0c5IFR/mak+Tq1rcGqdWo0e48XtpP8Nn3f5mpR/Huu8y89P2cnvg5QZ7hTfq1TePyPfu+V6qdL7Zvh5OEP5ZPj91pJJYcBgvvKeas0U35sj3/etmRRDheSk45e/VA2fskHabPiLFBR/hwBG/mqm7OSfy88d7d93bluG/sE8f/nEbgDOnNPC79/wXn7883f44nefI50y/mr9B7jh4m7mt2f588d3c/b8Nv7LmotqH3ASrN4tF83sz4FH3P0fzGwd0O7uf1uvDnDlRMtqHP9m4ObSj8uByc7TPBPQHegrdD0qdC0qdC0qmuFavMfdu2r9RUjXUCewr/T4EHBVYJ3JlJ3E3TcBmwLaGsTMdrr7ykYdb7bT9ajQtajQtaho9msR0j9yjOI3d4A5ozynVp3JlImIyDQJ+dDdBawuPV4B7AmsM5kyERGZJiFdQw8C281sMXAjsM7MNrr7hjHqXENxXGiiZVOtYd1MTULXo0LXokLXoqKpr0XdwWIAM1sIrAGecvea92esVWcyZSIiMj2CAoGIiDQvDcyKRMTMzjCzNWZ25ky3RU4f0QUCM9tsZjvMbEP92rOXmXWb2fYRP59y3pMpmy3MbL6ZPWxmW83sATNrifhaLAR+BFwNPGFmXbFei7LS78lzpcfRXouoAsHIFdDABaWVzU2n9Av/LYprNGqe92TKZuasJuwzwJ+5+0eA/cA64r0WlwNfdvc/Ah4FbiDea1H2J0B75L8jcW06B/QAW0qPt1Kctjpl21nMoALwKeCHpZ97OPW8r5xE2ay5Zu5+34gfu4B/C9xb+jm2a/EkgJldRzErOINI3xcAZnYD0E/xC0IPEV+LqDICTl3F3D2DbZky7t7n7kdHFNU678mUzTpmtgpYCPyKiK+FmRnFLwmHKU7djvJamFkL8IfAbaWiqH9HYgsEsa5ijnrlt5mdAfwF8DkivxZedAvwIvBB4r0WtwH3ufuR0s9Rvy9mXYMnKdZVzNGu/C598/s74HZ3f5O4r8WtZra+9OMC4KtEei2ADwO3mNk24Arg48R7LaIbI6i1AjoGs23ldyP9HsWNDL9iZl8B/hb4d5Fei03AFjP7PPASxffFUzFeC3e/rvy4FAw+Qby/I/EtKLNIVzHXOu/JlM1muhYVuhYVMV+L6AKBiIicLLYxAhERqaJAICISOQUCEZHIKRCIiEROgUBEJHL/H/U5GPCCOESIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "# plt.style.use('seaborn')\n",
    "sns.kdeplot(train_df['text_len'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对新闻句子的统计可以得出，本次赛题给定的文本比较长，每个句子平均由907个字符构成，最短的句子长度为2，最长的句子长度为57921。\n",
    "\n",
    "下图将句子长度绘制了直方图，可见大部分句子的长度都几种在2000以内。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Histogram of char count')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAERCAYAAABxZrw0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAW9ElEQVR4nO3df5BdZ33f8fcHySJCio0cb0QEcXc8qElobIFHOBIjkoXagFw3dZwU0xhSwERT4jC06bTI2O04nnRCPalDagc3AgGuwW6BAHGrmMgGVItE/rEK+AdtqSkVJsJuBDYSasAp7rd/nLPW1f7Q3r3r3dWefb9m7uw5zzn3nOd5Rvrss885955UFZKk7nnOQldAkjQ3DHhJ6igDXpI6yoCXpI4y4CWpowx4SeooA14TJLkmyXt61vckeV3P+u8luXRhajf/2v54LMk3k5w14PuvmYOqSSdkwGvGquqdVfUf+9k3yfOT/OO5rtNcSfJ84DeAs4AXA48tbI0WRpLhJG9e6HpoZgx4zbXnA4s24Gnq/0RVfa+q/qqqvrfQFVogw8CbF7gOmiEDXjOW5MPjR3NJfrOdwngsya+1ZbcC9wM/nuTxJJ/p2f+fJXk0yVeSbO0pv749xq4kn03y9rb8QJJXJ/l8kpvHnfdge6w39ex7a3uc307yl0l+5QTteU6Sf9Me54EkL5+u/ic4zu+0x3kkyWt6Np+W5DNJvp3k93r239H221d795+qvSc491h/fiPJG0/Uz+OnjNpzDbflvztJPfcCnwRe0fbD+6erj04SVeXL13Ev4BrgKPB4+/pr4HU92z8MvLln/fR2nzXAjwB/2LNtGDgw7vjnAw+3+7+kPcda4KeAR4HlwG3A23recwD4c2AL8MNt2ZnA54BVwDrg8Z593wh8AvgA8OvAB0/Q3rcBnwV+CHh1+/7nTlX/ExznV4E72+O8AvhmT38eBl4OvBB4qu2zn2nruBzYBNx3ovae4LwXAA/S/LUxDHwXWH2Cfr4GuGbcuYanqme7zwiwZ6H/bfqa2csRvKZyY1W9oKpeAPzZNPseBr4C/C5N2PzDafbfCnykqp6sqv8K3Au8kiZQQhN4K5j4F+Z1VfWFqvouQFU9SjP980+Bj9KE15h9NL+kxn6e6N/6VuD9VfX9qvpc256zp2nDZF4HfKA9zp9V1bqebf+pqu6vqoM0QXtqVd0LXA9cC9wA/OiJ2jvNeT9aVd+pqgNV9cNVdZSp+3m8nKiefbZdJyEDXrNWVU/TjPo+Afwc8MUkK6Z727jlAr5DcxHzq8ApwEfGveee3pUkrwQ+BXyNifPDT4/7OZ3J6jMrSd6SZHW7+j/HnyvJZcDvA1+k+StjvHsmKevnvL+UZOyXywnblWQ5x/9inFBPLV4GvGYtyd+kmeL4LPAu4AU0UzUA3wZ+JMnz2tdK4A7gsvYOm5+kmar4AnARcG9Vvaiqfr6q/mqaU/8McB/NdM6Fs2jCHcDlSZ6b5OdopjoeHuA4fwK8tT3OTwPvAb7fbpssLDe37/kj4O8NcL7e8/5yktPaYL8R+H9M3c9HgB9v37sNeG7PsaYK9W8BL0qyLMmaJMtmUV/Nk+ULXQEtflX1P9oLcf+rLbqxqh5rt303yb+mGRk+B9hcVXcluYVm3vj7wFur6n8n+TxwQ5LX00yTfBZ4R1X9YIpTf4JmOuibwMeAo+0vm5n6IM38/9doguzvV9VTAxxnZ89xjgC/XFU/SDLV/h+mqfebaH5JDSU5raoOz+SkVbU7ycuAh4AfAP+kqh4HHp+in/8DcEeS3TRTWF/v4xwPJ7kLOEjzV9GLgaV6R9GikSr/CtPJIcn1wJ9X1UeSPI9mtPmWqnpggasmLUpO0ehk8sfAlUm+Cfw3mtHllxe2StLi5QhekjrKEbwkdZQBL0kdddLcRXPGGWfU8PDwQldDkhaV/fv3f6uqhibbdtIE/PDwMKOjowtdDUlaVJJMeZurUzSS1FEGvCR1lAEvSR1lwEtSRxnwktRRBrwkdZQBL0kdZcBLUkdNG/DtQwTuSLI7yaeSrEiyM8m+JFf37NdXmSRpfvQzgr8MuL6qXkPzjMY3AMuqajNwVpL1SS7pp2yuGjFmePuuuT6FJC0a035VQVW9r2d1iOZp9e9t13fTPPX9ZTRPppmu7JHeYyfZRvPIMM4888yBGiBJmlzfc/BJNgNrgG/QPLYL4AmaB/au6rPsOFW1o6o2VtXGoaFJvytHkjSgvgI+yenADcBbgaPAynbT6vYY/ZZJkuZJPxdZVwAfB66sqq8D+2mmWwA2AAdmUCZJmif9fF3w5cC5wFVJrgI+BLwpyTpgK7AJKGBvH2WSpHky7Qi+qm6qqjVVNdK+bgZGgHuAV1XV4ao60k/ZXDVCkjTRQA/8qKonOXaHzIzKJEnzwwufktRRBrwkdZQBL0kdZcBLUkcZ8JLUUQa8JHWUAS9JHWXAS1JHGfCS1FEGvCR1lAEvSR1lwEtSR3Uu4Ie37/LZrJJEBwNektQw4CWpowx4SeooA16SOqqvgE+yNsnedvntSfa0ry8l+YMky5M82lN+drvvziT7klw9l42QJE00bcAnWQPcDKyCZ57ROlJVI8Be4P3AOcBtPc9tfSjJJcCyqtoMnJVk/Zy1QpI0QT8j+KeBS4EjvYVJXgisrapRYBNwUZL72lH7cpoHbo89j3U3sGX8gZNsSzKaZPTQoUOzaIYkabxpA76qjlTV4Uk2XQHc1C7fD5xfVecBpwAX0oz4D7bbnwDWTnLsHVW1sao2Dg0NDVJ/SdIUBrrImuQ5wKuAPW3Rg1X1WLs8CqwHjgIr27LVg55LkjSYQUP3lcC9VVXt+i1JNiRZBlwMPADs59i0zAbgwGwqKkmameUDvu+1wN0969cCtwIBbq+qu5KcCuxNsg7YSjNPL0maJ30HfHvXzNjyu8dte5jmTpresiNJRoALgOummMeXJM2RQUfwfamqJzl2J40kaR554VOSOsqAl6SOMuAlqaMMeEnqKANekjrKgJekjjLgJamjDHhJ6igDXpI6yoCXpI4y4CWpowx4SeooA16SOsqAl6SOMuAlqaMMeEnqqL4CPsnaJHvb5eVJHk2yp32d3ZbvTLIvydU975tQJkmaH9MGfJI1wM3AqrboHOC2qhppXw8luQRYVlWbgbOSrJ+sbK4aIUmaqJ8R/NPApcCRdn0TcFGS+9oR+nJghGOP5tsNbJmi7DhJtiUZTTJ66NChgRshSZpo2oCvqiPjHph9P3B+VZ0HnAJcSDO6P9hufwJYO0XZ+GPvqKqNVbVxaGho8FZIkiYY5KHbD1bVU+3yKLAeOAqsbMtW0/zimKxMkjRPBgndW5JsSLIMuBh4ANjPsSmYDcCBKcokSfNkkBH8tcCtQIDbq+quJKcCe5OsA7bSzNPXJGWSpHnSd8BX1Uj782GaO2l6tx1JMgJcAFw3Nmc/WZkkaX4MMoKfVFU9ybG7ZqYskyTNDy98SlJHGfCS1FEGvCR1lAEvSR1lwEtSR3U24Ie372J4+66FroYkLZjOBrwkLXUGvCR1lAEvSR1lwEtSRxnwktRRBrwkdZQBL0kdZcBLUkcZ8JLUUQa8JHWUAS9JHdVXwCdZm2Rvu3xakjuS7E7yqSQrkixP8miSPe3r7HbfnUn2Jbl6LhshSZpo2kf2JVkD3AysaosuA66vqjuT3AS8DvgL4LaqelfP+y4BllXV5iQfTLK+qh559puAXyomSZPoZwT/NHApcASgqt5XVXe224aAvwQ2ARclua8dtS8HRjj2PNbdwJbxB06yLcloktFDhw7NriWSpONMG/BVdaSqDo8vT7IZWFNV9wD3A+dX1XnAKcCFNCP+g+3uTwBrJzn2jqraWFUbh4aGZtEMSdJ4007RTCbJ6cANwC+2RQ9W1VPt8iiwHjgKrGzLVuMFXUmaVzMO3SQrgI8DV1bV19viW5JsSLIMuBh4ANjPsWmZDcCB2VdXktSvQUbwlwPnAlcluQq4CbgWuBUIcHtV3ZXkVGBvknXAVpp5eknSPOk74KtqpP15E02oj3fOuP2PJBkBLgCum2weX5I0dwaag+9XVT3JsTtpJEnzyAufktRRBrwkdZQBL0kdZcBLUkcZ8JLUUQa8JHWUAS9JHWXAS1JHGfCS1FEGvCR1lAEvSR1lwEtSRxnwktRRBrwkdZQBL0kd1fmAH96+a6GrIEkLovMBL0lLVV8Bn2Rtkr096zuT7Ety9UzLJEnzY9qAT7IGuBlY1a5fAiyrqs3AWUnW91s2d82QJI3Xzwj+aeBS4Ei7PsKx56zuBrbMoOw4SbYlGU0yeujQoZnXXpI0pWkDvqqOVNXhnqJVwMF2+Qlg7QzKxh97R1VtrKqNQ0NDg7VAkjSpQS6yHgVWtsur22P0WyZJmieDhO5+jk23bAAOzKBMkjRPlg/wnk8De5OsA7YCm4Dqs0ySNE/6HsFX1Uj78wjNBdR7gFdV1eF+y57VmkuSTmiQETxV9STH7pCZUZkkaX544VOSOsqAl6SOMuAlqaMMeEnqKANekjrKgJekjjLgJamjDHhJ6igDXpI6yoCXpI4y4CWpowx4SeooA16SOsqAl6SOMuAlqaMMeEnqKANekjpqoCc6JXk7cGm7+nyaB2xfAHytLXtHVT2UZCfwEmBXVf3WbCsrSerfoI/suwm4CSDJDcAtwLeq6l1j+yS5BFhWVZuTfDDJ+qp65NmotCRperOaoknyQmAtsBG4KMl9SXYmWU7zwO2x57HuBrZM8v5tSUaTjB46dGg2VZEkjTPbOfgraEby9wPnV9V5wCnAhcAq4GC73xM0vwiOU1U7qmpjVW0cGhqaZVWmNrx9F8Pbd83Z8SXpZDRwwCd5DvAqYA/wYFU91m4aBdYDR4GVbdnq2ZxLkjRzswndVwL3VlUBtyTZkGQZcDHwAM2F17FpmQ3AgdlUVJI0MwNdZG29Fri7Xb4WuBUIcHtV3ZXkVGBvknXAVmDTrGoqSZqRgQO+qt7ds/wwcM647UeSjNDcPnldVR0e9FySpJmbzQh+WlX1JMfupJEkzSMvfEpSRxnwktRRBrwkdZQBL0kdZcBLUkcZ8JLUUQa8JHWUAS9JHWXAS1JHGfCS1FEGvCR1lAEvSR1lwEtSRxnwktRRBrwkdZQBL0kdNeOAT7I8yaNJ9rSvs5PsTLIvydU9+00okyTNn0FG8OcAt1XVSFWNAOuBZVW1GTgryfokl4wve/aqLEnqxyCP7NsEXJTkVcBDwFMceyzfbmAL8LJJyh4Zf6Ak24BtAGeeeeYAVZEkTWWQEfz9wPlVdR5wCrAVONhuewJYC6yapGyCqtpRVRurauPQ0NAAVZEkTWWQgH+wqh5rl0eBM4CV7frq9phHJylbcMPbdy10FSRp3gwSvLck2ZBkGXAxcAXNFAzABuAAsH+SMknSPBpkDv5a4FYgwO3Ap4G9SdbRTNdsAmqSMknSPJpxwFfVwzR30jwjyQhwAXBdVR2eqkySNH8GGcFPUFVPcuyumSnLJEnz56S4+ClJevYZ8JLUUQa8JHWUAS9JHWXAS1JHGfCS1FEGvCR1lAEvSR1lwEtSRxnwktRRBrwkdZQBL0kdZcBLUkcZ8JLUUQa8JHXUkgv44e27fDarpCVhyQW8JC0VAwV8ktOS3JFkd5JPJVmR5NEke9rX2e1+O5PsS3L1s1ttSdJ0Bh3BXwZcX1WvAR4HtgO3VdVI+3ooySXAsqraDJyVZP2zVGdJUh8GCviqel9V3dmuDgE/AC5Kcl87al8OjHDsmay7gS3jj5NkW5LRJKOHDh0apCqSpCnMag4+yWZgDXAncH5VnQecAlwIrAIOtrs+Aawd//6q2lFVG6tq49DQ0GyqIkkaZ/mgb0xyOnAD8IvA41X1VLtpFFgPHAVWtmWr8YKuJM2rQS+yrgA+DlxZVV8HbkmyIcky4GLgAWA/x6ZlNgAHZl9dSVK/Bh1VXw6cC1yVZA/wZeAW4EvAvqq6C/g08KYk1wOvB06qm8+9F15S1w00RVNVNwE3jSv+zXH7HEkyAlwAXFdVhweqoSRpIAPPwfejqp7k2J00kqR55IVPSeooA16SOsqAl6SOWtIB7zdLSuqyJR3wktRlBjzeEy+pmwx4SeooA16SOsqAl6SOMuBb3lEjqWsMeEnqKAN+HEfxkrrCgJekjjLgJ+F8vKQuMOAlqaPm9PvgF7veUfyB9/ydBayJJM2cI/gZcOpG0mIy5yP4JDuBlwC7quq35vp8c6U32MeP7MfWHeVLOpnMacAnuQRYVlWbk3wwyfqqemQuzznfpgr+foz9QhjevstfDpKedamquTt48m+Bz1TVHyd5A7Cyqj7Us30bsK1d/QngKwOe6gzgW7OqbLfYHxPZJ8ezP463mPvjb1TV0GQb5nqKZhVwsF1+Aji3d2NV7QB2zPYkSUarauNsj9MV9sdE9snx7I/jdbU/5voi61FgZbu8eh7OJ0lqzXXg7ge2tMsbgANzfD5JUmuup2g+DexNsg7YCmyao/PMepqnY+yPieyT49kfx+tkf8zpRVaAJGuAC4C7q+rxOT2ZJOkZcx7wkqSF4UVPqSOSnJ7kgiRnLHRddHJY9AGfZGeSfUmuXui6zLUka5Ps7Vmf0PbZlC0WSU5LckeS3Uk+lWTFUu2LMe1U6H8GzgM+n2RoqfcJPPN/5ovt8pLrj0Ud8L2flAXOSrJ+oes0V9r/wDfTfLZg0rbPpmxhWjWwy4Drq+o1wOPAG1i6fTHmHOA3qupfAX8CvBr7BOB3gJVL9f/LYv82yRHgY+3ybppbMjv1VQg9ngYuBf6oXR9hYttfNouyRdNvVfW+ntUh4I3Ae9v1JdUXY6rqvwAk+VmaUfzpLNF/H2OSvBr4PzSDgBGWYH8s6hE8Ez8pu3YB6zKnqupIVR3uKZqs7bMpW3SSbAbWAN9gifcFQJLQDAKeBIol3CdJVgD/AtjeFi3J/y+LPeCX8idlJ2v7bMoWlSSnAzcAb2WJ98WYalwBPAi8gqXdJ9uB91XVd9r1JflvZNFUdApL+ZOyk7V9NmWLRjs6+zhwZVV9nSXcF2OSvCvJr7Srzwfew9Luk/OBK5LsAV4K/F2WYH8s9jn4+fqk7MlosrbXLMoWk8tpvrjuqiRXAR8C3rRE+2LMDuBjSd4GPEzz7+PupdonVfWzY8ttyP88S/D/y6L/oFOW8CdlJ2v7bMoWM/tiIvvkeEuxPxZ9wEuSJrfY5+AlSVMw4CWpowx4SeooA14nlSS/nmRPku+1P39hhu9/aZKXDnDea5KMzPR982nQtmnpWuy3SapjqupG4MYkX62qkQEOMRaAX3r2anXS6HLbNAe8i0YnpTbgX9wuPw/498CPAg9V1RVJtgDvpPlo/p8Crwd+DRgb8R+sqr89xbF/CPgw8CLgO+17/zlwCvBK4FTgdTSfYPwEzUfVv1pVb2nfvwe4Hzinql47g3M83ZatA/4CeAvwbmBPVe1J8ub27cOT1OWd/bRN6uUUjRaDbcDD7YdXfizJOVX1BZoAvhH4dFV9o6qupPkE53umCcBtwANVtQX4Q+Cn2/IXt+f4JM23Mf4YzdchnA8MJxn7DpJNwL6pwv0E5/jVth0/R/NlVW89wfuPq8sM2iY9w4DXYvATwC+0I+ezgBe25b9PE5p/MMPj/SRwX7v8YZrRODR/JQA8CqwA/i/wNuCjNN/OOPZ9JA9X1ScHOMdLgHvbsnuAnxr3npU9y+PrIs2YAa/F4CvAe9s5+atpQo92+bdpvjVwzPeA58Ez3644mf8OvLxdfjdNiEPz1bK9LqeZovkH47Yd7aPOk53jyxz7mPumdv2vab7yGJqpmDHj6wL9tU16hgGvxeD9wNYkdwP/CPhGkl8CvllV/xL4W0nObfe9E7gkyZ/SzGFPdbxz278IzgVumWK/O4Ergc+16y+cYr9+z/GBtq53A+tpRva3A+9I8u+Ab09zzH7aJj3Di6yS1FGO4CWpowx4SeooA16SOsqAl6SOMuAlqaMMeEnqqP8PbU91Acwm1FwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = plt.hist(train_df['text_len'], bins=200)\n",
    "plt.xlabel('Text char count')\n",
    "plt.title(\"Histogram of char count\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **新闻类别分布**\n",
    "\n",
    "接下来可以对数据集的类别进行分布统计，具体统计每类新闻的样本个数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'category')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAETCAYAAAA8rh0/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAXF0lEQVR4nO3de7SddX3n8feHBMaYEC5yJhoVs6CprQ5EbWSSGdSDglzrIA5Dq3VaR01rrbPGdqkgjKPUThmsl9aWtBnxRgUHbwxT1EbUSMoE4QTlUlvvEaWgUZCYWdYq/c4fzxNzPJyTc3LO3ifkx/u11l559m8/+/v8npPks3/nt5/926kqJEntOWBfd0CSNBwGvCQ1yoCXpEYZ8JLUKANekhplwEtSowx4zYskr09y67j7m5L8xj7s0q5+vD7J6/d1P6RhMOA1n45JsnZfd6J1SQ5N8l/2dT+07xnwmk87gN/c1514CDgUMOBlwGtefQD4d0kOG9+Y5EVJvpbkriQv7du+kmR5kg/30yhnJLmyf+wNSf6h3/+3pztoklcluSPJN5P82gz2/61+/zuTvGZc+2/2Nb6T5I3Tte+h/gvGne+rxrX/WpKvJ9m2a/oqyW8kefe4fTYlGe3b39ffvpfkQ+lcDtwEPDbJ3Uk+Pl1/1K6F+7oDekj5LvBXwAt3NSR5IvC7wGrgQODmJFcDNwOPB34AHEk3+r85yeHAecAyugHKBuCSqQ6Y5KT+eMfSjWxvS3JVVe2cYv+H9fuvAb4PfCPJn/X7vwn4N8BXgMuSHFxVP9hD+2T1fxG4CFgL/LDvz9VAxrXfD9yQZOtU59V7HnA23W9FXwGeVFXPT7IC2FRVK6Z5vhpnwGu+/TldKG/v7z8TOAr4Qn9/EV2w3wysAn4MPKxv+xBwH/BF4K3Ax4Ffn+Z4pwDvq6rv0wX2wXvauar+Mcl/pAv5pwGHA0cAO4G/Af4AuAp42bgQn6p9MicCf1VV3+rvLwdI8oq+/Zv9/Y8AJ9O9KI6XcdtjVfV/+v2/CByyp3PTQ49TNJpXVXU93Qj1yX1TgPdW1SOr6pHAY4Ab6AL+l4Gv9fs9Abi5qu4Hngp8EHgG8LkkB830+En+fZLle3j8aOA64B7g94Bvjnv4OcCf0L3Y3J5kZJr2mfTn2Uke398dv/JfTbi/y6PHbX91wv7SzzDgtS+sB5b2258CTk3yyCQHA7fQhflW4ATgS8A3gCOr6rtJfh74ZH97DfBI4BF7ONZfA89Pckgf7H8K/PMe9n8ysA14J11gPwYgycOB2+leeF5HN6L/uana91D/k8Dp/fsLS/v+LAY+AZyR5NFJHgU8F9hINzX12L4PpwFHj6s1Vah/D3hEkof3t0V76I8a5hSN9oW/BC4GqKrbk/w+sIXu3+PbqurzAEm+SRfwi4HP9ft/Kclm4Ot9rT+tqrumOlBVbUzyZOA24CfAK6vq7j307VrglcC36QL268DPV9W2JJf0dQ4EPgrcWFX3T9a+h/58IckFwPXAAuCtVXVzf77n0U33BPhvVXVbkq8Cv5tkE90Lyd/soe+7jvGDJP+DboR/AN28/tf2/Cy1KK4HL0ltcopGkhplwEtSowx4SWqUAS9JjXrQXEVzxBFH1IoVK/Z1NyRpv7J169bvVtWkn7140AT8ihUrGBsb29fdkKT9SpJvTPWYUzSS1CgDXpIaZcBLUqMMeElqlAEvSY0y4CWpUQa8JDXKgJekRhnwktSoB80nWaey4txr9mr/bRedPqSeSNL+xRG8JDXqQT+CHzZ/Q5DUKkfwktQoA16SGmXAS1KjDHhJapQBL0mNMuAlqVEGvCQ1yoCXpEYZ8JLUqBkFfJJlSTb32y9Lsqm/fT7JXyRZmOSOce3H9PtemmRLkguGeRKSpAeaNuCTHAa8B1gMUFXrq2q0qkaBzcD/BI4FrtjVXlW3JTkLWFBVa4Gjkqwc2llIkh5gJiP4+4FzgB3jG5M8GlhWVWPAGuCMJDf2o/aFwChwZb/7RuD4gfVakjStaQO+qnZU1X2TPPRyYH2/fRNwYlUdBxwInEY34r+zf/weYNnEAknWJRlLMrZ9+/bZ9F+SNIVZvcma5ADgBGBT33RrVd3Vb48BK4GdwKK+bclkx6qqDVW1uqpWj4yMzKYrkqQpzPYqmqcBn62q6u9flmRVkgXAmcAtwFZ2T8usArbNpaOSpL0z2/XgTwauG3f/QuByIMDVVXVtkqXA5iTLgVPp5uklSfNkxgHfXzWza/u1Ex67ne5KmvFtO5KMAicBF08xjy9JGpKhfqNTVd3L7itpJEnzyE+ySlKjDHhJapQBL0mNMuAlqVEGvCQ1yoCXpEYZ8JLUKANekhplwEtSowx4SWqUAS9JjTLgJalRBrwkNcqAl6RGDXW5YMGKc6/Zq/23XXT6kHoi6aHGEbwkNcqAl6RGGfCS1CgDXpIaNaOAT7IsyeZ+e2GSO5Js6m/H9O2XJtmS5IJxz3tAmyRpfkwb8EkOA94DLO6bjgWuqKrR/nZbkrOABVW1FjgqycrJ2oZ1EpKkB5rJCP5+4BxgR39/DXBGkhv7EfpCYBS4sn98I3D8FG0/I8m6JGNJxrZv3z7rk5AkPdC0AV9VO6rqvnFNNwEnVtVxwIHAaXSj+zv7x+8Blk3RNrH2hqpaXVWrR0ZGZn8WkqQHmM0HnW6tqh/122PASmAnsKhvW0L3wjFZmyRpnswmdC9LsirJAuBM4BZgK7unYFYB26ZokyTNk9mM4C8ELgcCXF1V1yZZCmxOshw4lW6eviZpkyTNkxkHfFWN9n/eTnclzfjHdiQZBU4CLt41Zz9ZmyRpfgxssbGqupfdV81M2SZJmh++8SlJjTLgJalRBrwkNcqAl6RGGfCS1CgDXpIaZcBLUqMMeElqlAEvSY0y4CWpUQa8JDXKgJekRhnwktQoA16SGmXAS1KjDHhJapQBL0mNMuAlqVEzCvgky5Js7rcPSfKxJBuTfCTJQUkWJrkjyab+dky/76VJtiS5YJgnIUl6oGkDPslhwHuAxX3TC4C3VNWzgbuBU+i+hPuKqhrtb7clOQtYUFVrgaOSrBzOKUiSJjOTEfz9wDnADoCquqSqPtE/NgJ8B1gDnJHkxn7UvhAYZfcXbm8Ejp9YOMm6JGNJxrZv3z63M5Ek/YxpA76qdlTVfRPbk6wFDquqG4CbgBOr6jjgQOA0uhH/nf3u9wDLJqm9oapWV9XqkZGROZyGJGmihbN5UpLDgbcDz+ubbq2qH/XbY8BKYCewqG9bgm/oStK82uvQTXIQ8AHgvKr6Rt98WZJVSRYAZwK3AFvZPS2zCtg29+5KkmZqNiP4FwNPAc5Pcj6wHrgQuBwIcHVVXZtkKbA5yXLgVLp5eknSPJlxwFfVaP/nerpQn+jYCfvvSDIKnARcPNk8viRpeGY1Bz9TVXUvu6+kkSTNI9/4lKRGGfCS1CgDXpIaZcBLUqMMeElqlAEvSY0y4CWpUQa8JDXKgJekRhnwktQoA16SGmXAS1KjhrrYmIZvxbnX7NX+2y46fUg9kfRg4whekhplwEtSowx4SWqUAS9JjTLgJalRBrwkNWpGAZ9kWZLN4+5fmmRLkgv2tk2SND+mDfgkhwHvARb3988CFlTVWuCoJCtn2ja805AkTTSTEfz9wDnAjv7+KHBlv70ROH4v2n5GknVJxpKMbd++fe97L0ma0rQBX1U7quq+cU2LgTv77XuAZXvRNrH2hqpaXVWrR0ZGZncGkqRJzeZN1p3Aon57SV9jpm2SpHkym9Ddyu7pllXAtr1okyTNk9ksNnYVsDnJcuBUYA1QM2yTJM2TGY/gq2q0/3MH3RuoNwAnVNV9M20baM8lSXs0q+WCq+pedl8hs1dtkqT54RufktQoA16SGmXAS1KjDHhJapQBL0mNMuAlqVEGvCQ1yoCXpEYZ8JLUKANekhplwEtSowx4SWqUAS9JjTLgJalRBrwkNcqAl6RGzeoLP/TQseLca/Zq/20XnT6knkjaW47gJalRsxrBJ3kZcE5/91BgK3AS8LW+7RVVdVuSS4EnANdU1Rvn2llJ0szN9jtZ1wPrAZK8HbgM+G5VvWbXPknOAhZU1dok70yysqq+PIhOS5KmN6cpmiSPBpYBq4EzktyY5NIkC4FRdn/h9kbg+Emevy7JWJKx7du3z6UrkqQJ5joH/3K6kfxNwIlVdRxwIHAasBi4s9/vHroXgp9RVRuqanVVrR4ZGZljVyRJ48064JMcAJwAbAJuraq7+ofGgJXATmBR37ZkLseSJO29uYTu04DPVlUBlyVZlWQBcCZwC90br7umZVYB2+bSUUnS3pnLdfAnA9f12xcClwMBrq6qa5MsBTYnWQ6cCqyZU08lSXtl1gFfVa8dt307cOyEx3ckGaW7fPLiqrpvtseSJO29oX6StaruZfeVNJKkeeQbn5LUKANekhplwEtSowx4SWqUAS9JjTLgJalRBrwkNcqAl6RG+ZV92qf8SkBpeBzBS1KjDHhJapQBL0mNMuAlqVEGvCQ1yoCXpEYZ8JLUKANekhplwEtSowx4SWrUXgd8koVJ7kiyqb8dk+TSJFuSXDBuvwe0SZLmz2zWojkWuKKqXgOQ5CxgQVWtTfLOJCuBYya2VdWXB9hvaUZc60YPZbMJ+DXAGUlOAG4DfgRc2T+2ETgeePIkbQ8I+CTrgHUARx555Cy6Ikmaymzm4G8CTqyq44ADgVOBO/vH7gGWAYsnaXuAqtpQVauravXIyMgsuiJJmspsAv7Wqrqr3x4DjgAW9feX9DV3TtImSZpHswney5KsSrIAOBN4Od0UDMAqYBuwdZI2SdI8ms0c/IXA5UCAq4GrgM1JltNN16wBapI2SdI82uuAr6rb6a6k+akko8BJwMVVdd9UbZKk+TOQr+yrqnvZfdXMlG2SpPnjm5+S1Ci/dFuaAz9IpQczA156kPLFQ3PlFI0kNcqAl6RGGfCS1CgDXpIaZcBLUqMMeElqlAEvSY0y4CWpUQa8JDXKgJekRhnwktQo16KRHqJc66Z9juAlqVEGvCQ1yoCXpEbNKuCTHJLkY0k2JvlIkoOS3JFkU387pt/v0iRbklww2G5LkqYz2xH8C4C3VNWzgbuBc4Erqmq0v92W5CxgQVWtBY5KsnJAfZYkzcCsAr6qLqmqT/R3R4CfAGckubEftS8ERtn9pdsbgeMn1kmyLslYkrHt27fPpiuSpCnM6TLJJGuBw4BPAO+qqruSvBc4DVgM3Nnveg/wlInPr6oNwAaA1atX11z6IunBxcsw971ZB3ySw4G3A88D7q6qH/UPjQErgZ3Aor5tCb6hK0nzarZvsh4EfAA4r6q+AVyWZFWSBcCZwC3AVnZPy6wCts29u5KkmZrtCP7FdFMu5yc5H/g0cBkQ4OqqujbJUmBzkuXAqcCaQXRYkjQzswr4qloPrJ/Q/IYJ++xIMgqcBFxcVffNqoeSpFkZ6lo0VXUvu6+kkSTNI9/4lKRGGfCS1CgDXpIa5XrwkvZLfpBqeo7gJalRBrwkNcqAl6RGGfCS1CgDXpIaZcBLUqMMeElqlNfBS9IkWrjO3hG8JDXKgJekRhnwktQo5+AlaR+Yjzl+R/CS1CgDXpIaZcBLUqOGHvBJLk2yJckFwz6WJGm3oQZ8krOABVW1FjgqycphHk+StFuqanjFkz8BPl5VH03yK8CiqnrXuMfXAev6u48HvrgX5Y8Avjuwzlrf+vtP/f2579YffP3HVdXIZA8M+zLJxcCd/fY9wFPGP1hVG4ANsymcZKyqVs+te9a3/v5Xf3/uu/Xnt/6w5+B3Aov67SXzcDxJUm/YgbsVOL7fXgVsG/LxJEm9YU/RXAVsTrIcOBVYM8Das5rasb71G6i/P/fd+vNYf6hvsgIkOQw4Cbiuqu4e6sEkST819ICXJO0bvukpSY0y4CWpUftVwCc5OsmvJ3l1knOT/EqSg/d1v6aTZGGSM5IcN6H97AHVf0SSk5IsSfIvkpyd5KRB1N7DMQf3RlDyxP7PA/qf0yuTPHNQ9Scc63eSfDDJm5M8akA11yd5yvR7zukYT0kykmRBkpckeXGShw+o9tOT/PKg6unBY7+Zg0/yWuBoYCPdh6aW0F16eTZwQlV9Zx92b4+SfAj4NjACHAq8qKq+leRTVTWnIEvyCODTwF8DTwXuBW7rj3NAVf3OnDrfHeNm4GHArjfJAzwJ+Nxc+9/X/1RVPTPJO+iu7NoKnAlcX1WvG0D906vqmiTPAY4D/pLuZ/VbVfVvB1D/c8Bm4F8C66vqM3OtOaH+emAp8FjgC3Sf+P7XwKFVdcoca7+pr/sd4GTg3cBbq+of51J3imMdTXfZ9DK6weU24Jqq+sGgjzVISRYCpwDfqaobx7WfXVUfGED9R9B9CHQL8GPgOcD3q+oTc669HwX89ZP9Z0zyZmBLVX1wjvWvAx4O7BjfDNQAQvjju/4jJlkL/DFwHnD+AGqfCDy1qv4wydOB/7Ar1JNsqqrRudTv6ywD/ojug2uvqaodST5dVSfMtXZff1fA31BVa/q2BcDNVbVqAPX/jC5UtgEX7xoMJPlMVT1jAPV39X858HJgFPgs3ZVjVw2g/o1VdVySQ4BXVNUb+5/PvVW1dI61f/ozSHIG8BK6wH9LVb1vrn0fdxwHaJPXHuoAbX/6Rqe/S/JO4Eq65Q8WAU8DnglcOID6Z9ONXs6pqh3T7Lu3/jnJs6rqk1W1JckpdOcx5/CiG+2+Icknq+o64DqAJC8E/mkA9amqbwMvTHIC8JEklwCDHBkcneQPgcVJlvXHe+KgilfVy5M8je764hX99NLRdP9pByH9cf4BOD9J6P5tnkz3WZC52p7kV4F/BTwuyRLgWOCOAdT+QZKnAp+j+63gTcDngVcNoPZ4p08yQPtIP8X6dGDWA7RhDs56i6vqt/tjrQU+nOS8AdQFeDJwxYQB2uv6Y22ac/Wq2m9uwHPpRr/vAN4GvBA4eID1DwWWDKHfS4GXTWg7kG6KYFD9fvqEtlfT/Qo/6HM5EPivwGcGWHNB/w/9ZcAvAQcB/wt44oD7vhD4T8Dbgf88qL/rQf09TvP3+wrg+cCRwKXAh+h+c5tr7RX9z/pm4LVDPId3AO+km+o4hm6q7PfoXlgOmWPtZcDHgKVD6vtHgWeNu384cC3wvQHUPgy4HjhuQvsLgY1zrb/fTNFI2r8leS7d9NViuum+rcBVNYA5+CSHAj+pqp1zrTVJ7aXAC6pq/bi2A4EXV9WfD6D+ocCx1f0Gvqvt1cCGqvr+nGob8JLUpv1pDl7SfmrIFzEMdQ5+f67vCF7S0PVXYr2bIVzEMMza+3t9A17SvBjyPPnQau/P9Q14SWrUfrVUgSRp5gx4SWqUAa+HtCRPSvKkfd0PaRgMeD3UPam/Sc3xTVY1KcnD6C49ewzwfbqP+V9J9ynKr1TVi/r1b57bP+XOqnpWv2Tue+lWhrytunVsFgEfpvuI+leB24E39/WXA9+iW4Dqn/r1Q26i+2TiyUneAPxdVb0/yeuBv6+q9w/9ByDhCF7tWgfcUlXH063b8ot0a9CcSLfg2LKqOg+4CLioqp417nm3V9XTgUclORb4BboQPx74uar678BL+/2eAXyZbo0b6L5YfktVndzffy/diwt0i4/976GdsTSBAa9W/QKwa+3ud9Otd/4S4H10I/FFUzzv8cBz+5H4UcCj6VYv/SW6lTr/uN/vCXRLAgPcQPcCAl3of3hXsar6KnBwktH+sR/O8bykGTPg1aq/p1tfG+C1wP+lW5L2V4H/N26/H9J9TJx+md8vAm+rbh39C+iW5D0F+P2qWlu710j/W7rROv2ff9tvT/ZBlffTraT43jmflbQXnINXk/p58/fQzaV/j27kfQndFyosAF5VVdcnOZxubn4R3ZewbAXeBTySbm2Q5wOPo1uO9kt0vwm8kW5a5t10I/xvMm4OviZ8yUr/pQ430k3v+B9O88aAl6aR5KV0I/8f97c/qqpNM3zuE+leMP6iqi4dWielSRjwktQo5+AlqVEGvCQ1yoCXpEYZ8JLUKANekhr1/wF8GfYXlJz+gAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_df['label'].value_counts().plot(kind='bar')\n",
    "plt.title('News class count')\n",
    "plt.xlabel(\"category\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在数据集中标签的对应的关系如下：{'科技': 0, '股票': 1, '体育': 2, '娱乐': 3, '时政': 4, '社会': 5, '教育': 6, '财经': 7, '家居': 8, '游戏': 9, '房产': 10, '时尚': 11, '彩票': 12, '星座': 13}\n",
    "\n",
    "从统计结果可以看出，赛题的数据集类别分布存在较为不均匀的情况。在训练集中科技类新闻最多，其次是股票类新闻，最少的新闻是星座新闻。\n",
    "\n",
    "#### **字符分布统计**\n",
    "\n",
    "接下来可以统计每个字符出现的次数，首先可以将训练集中所有的句子进行拼接进而划分为字符，并统计每个字符的个数。\n",
    "\n",
    "从统计结果中可以看出，在训练集中总共包括6869个字，其中编号3750的字出现的次数最多，编号3133的字出现的次数最少。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5340\n",
      "('3750', 373091)\n",
      "('6577', 1)\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "all_lines = ' '.join(list(train_df['text']))\n",
    "word_count = Counter(all_lines.split(\" \"))\n",
    "word_count = sorted(word_count.items(), key=lambda d:d[1], reverse = True)  # ascending order\n",
    "\n",
    "print(len(word_count))\n",
    "print(word_count[0])\n",
    "print(word_count[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里还可以根据字在每个句子的出现情况，反推出标点符号。下面代码统计了不同字符在句子中出现的次数，其中字符3750，字符900和字符648在20w新闻的覆盖率接近99%，很有可能是标点符号。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('3750', 9913)\n",
      "('900', 9898)\n",
      "('648', 9628)\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "train_df['text_unique'] = train_df['text'].apply(lambda x: ' '.join(list(set(x.split(' ')))))\n",
    "all_lines = ' '.join(list(train_df['text_unique']))\n",
    "word_count = Counter(all_lines.split(\" \"))\n",
    "word_count = sorted(word_count.items(), key=lambda d:int(d[1]), reverse = True)\n",
    "\n",
    "print(word_count[0])\n",
    "print(word_count[1])\n",
    "print(word_count[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **数据分析的结论**\n",
    "\n",
    "通过上述分析我们可以得出以下结论：\n",
    "\n",
    "1. 赛题中每个新闻包含的字符个数平均为1000个，还有一些新闻字符较长；\n",
    "2. 赛题中新闻类别分布不均匀，科技类新闻样本量接近4w，星座类新闻样本量不到1k；\n",
    "3. 赛题总共包括7000-8000个字符；\n",
    "\n",
    "通过数据分析，我们还可以得出以下结论：\n",
    "\n",
    "1. 每个新闻平均字符个数较多，可能需要截断；\n",
    "\n",
    "2. 由于类别不均衡，会严重影响模型的精度；\n",
    "\n",
    "### **本章小结**\n",
    "\n",
    "本章对赛题数据进行读取，并新闻句子长度、类别和字符进行了可视化分析。\n",
    "\n",
    "### **本章作业**\n",
    "\n",
    "1. 假设字符3750，字符900和字符648是句子的标点符号，请分析赛题每篇新闻平均由多少个句子构成？\n",
    "2. 统计每类新闻中出现次数对多的字符"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f5d67de36a0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD2CAYAAAA6eVf+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3daXRc5Z3n8e+/Ni0lWZYl2bKwQTZ2WAw4JI5jDksLJhBI4k4fZiYx6SSTbZjukPScM2cmJ0mbE6aHSdKZnLzo5JBud8gyCR3amemQhbhDZhoTwpjFBpwFjDFgwAbZ8oJtSZZqe+ZFVUm2VCXdkmq99/c5x4fy9XNv/a9U/PTouc99rjnnEBGRYAjVugAREakehb6ISIAo9EVEAkShLyISIAp9EZEAidS6gJl0d3e7/v7+WpchItJQdu3adcQ511Po3+o69Pv7+9m5c2etyxARaShm9nKxf9PwjohIgCj0RUQCRKEvIhIgdT2mLyLBk0wmOXDgAGNjY7Uupe41NzezbNkyotGo530U+iJSVw4cOEB7ezv9/f2YWa3LqVvOOY4ePcqBAwdYsWKF5/00vCMidWVsbIyuri4F/izMjK6urpJ/I1Loi0jdUeB7M5evk0JfRCRAPIW+md1tZjvMbHMpbcxsiZk9XOqxquVz//Q7Pv3Dp2pdhohI1cwa+mZ2MxB2zl0BrDSz1V7amFkn8D0gXsqxqumZ109y/29f4/ApzRIQkWDwMntnANiae/0AcBXwvIc2/xt4P/CTUo5lZrcCtwKce+65Hsqbu9HxFBkHP9v9Oh+/yvvVbxGpjv/6sz/wzGsny3rMi/sW8IWNa8p6zEbiJfTjwMHc62PAW7y0cc6dhGkXGmY9lnNuC7AFYN26dRV9luPIeAqAnzx9UKEvIhPuuOMOHnvsMUZHR+np6eHGG28kEonwkY98hO3bt7N9+3b6+/vZvXs3u3fvZnBwkK1bt7JmzRo+9alP8fTTTxONRrn33nvp7e2ddvzvfve70/bNrzN25nts376dvr4+Dh06BMD69ev50pe+NK9z8xL6w0BL7nUbhYeEvLQppV1VjCTSNEdD/PbACV4YGub8nrZaliMiU9SyR3711Vfz+c9/nttuu40TJ07Q1dU1rc0TTzzBgw8+yA9/+EN++tOf8uKLL5JKpXjkkUf4+c9/zq5du3j3u99d8PhT9+3r6yvY7otf/CI33HADTz75JBs3bpz3eXkJ3V1kh2EA1gL759imlHZVMZpI8a5Ll2IGP3nq4Ow7iEhgvPWtbwXgsssuO2v76dOnJ17fcsstRKNRzj33XBKJBHv27GH9+vUAvOc97+Gmm24qevyp+xZ7j/7+fvr6+mhra8O5+Q9+eAn9+4APmdnXgPcBfzCzO2dpc7/HYxVrV3GJVIZk2rGyO86V53dz39OvleULKiL+8PjjjwPw1FNP0d3dzdDQEADbtm2baBOPx8/a58ILL+SJJ54A4J577uH2228vevyp+8ZisYLvUW6zDu84506a2QBwPfAV59wgsHuWNifO+LcBL+2qbTSRHc9vjUV416VL+fyPf8dLR0ZYqSEeESE7/DIwMEBvby/veMc7uOWWW9i/f/+M+2zcuJFt27ZxzTXX0Nrayve//33P73fdddfxvve9b9b3mC+r597tunXrXKUeonLwjdNc+eV/4a//9aWcs7CVD979GP946wbevnL6uJ2IVM+zzz7LRRddVNMa7rjjDgYGBhgYGKhpHV4U+nqZ2S7n3LpC7QO74Nro+GRPv6stBsDRkcRMu4hIQNxxxx1lO9amTZsYHBw8a9u2bdtoaWkpskdlBTb0RxJpAOJN4cnQHx6vZUkikuOc8836O/fee2/Fjj2XkZrArr1zZk9/UWs29I8Mq6cvUmvNzc0cPXpUEytmkV9aubm5uaT91NOPRYiEQ3S2Rjk6op6+SK0tW7aMAwcOTMxkkeLyD1EpRWBDf2L2TlMYgK62Jo6qpy9Sc9FotKSHgkhpAju8MzI+2dMH6IrHFPoi4nuBDf18Tz+e6+l3tzVxRMM7IuJzgQ39fE+/Nd/Tb1NPX0T8L7ChP5pI0RwNEQ5lp4Utisc4cTpJMp2pcWUiIpUT2NAfSaQmxvMheyEX4Lhu0BIRHwts6I+Opydm7gB0xzVXX0T8L7ChX6ynr7n6IuJngQ390USa1thkT39yKQb19EXEvwIb+iPjKeJNkz397ni2p39E6++IiI8FOPTP7ukvaIkQCZlW2hQRXwtu6E8Z0zez3Fx99fRFxL8CG/qjibNn7wB0xbX+joj4W2BDf2T87J4+ZC/mHtHwjoj4WCBDP5XOMJ7KTCzBkNcVj3FMUzZFxMcCGfqjycmnZp1JyyuLiN8FM/SnLLaW19UWYzSRnliBU0TEbwIZ+iNTllXOy8/VV29fRPwqkKE/U08f0Fx9EfGtQIb+RE8/Nn1MH9BcfRHxrUCG/uTzcafP3gEN74iIfwUy9Cefjzu1p59bXlnTNkXEpwIZ+pPPxz27p98ai9ASDaunLyK+FcjQn+zpR6b9W1dbjGO6kCsiPhXI0M/39FumDO9A9mKullcWEb8KZOiPJNLEwiFikemn3x2PaXhHRHwrkKE/Op6atsJmXldbTI9MFBHfCmTojyTSBcfzYXL9HedclasSEak8T6FvZneb2Q4z21xKm6nbzKzTzH5hZjvN7O/mX/7cjCZSZz0160xd8RipjOPkaa2/IyL+M2vom9nNQNg5dwWw0sxWe2lTZL8PAfc459YB7Wa2rqxn49HweHrajVl53bm7cjVXX0T8yEtPfwDYmnv9AHCVxzaFth0FLjGzhcBy4NWpBzKzW3O/CewcGhrydBKlGh1PTbsxK29i/R1dzBURH/IS+nHgYO71MWCJxzaFtv0GOA/4C+DZ3PazOOe2OOfWOefW9fT0eDyN0owk0tMWW8vrimv9HRHxLy+hPwy05F63FdmnUJtC274A/Jlz7q+APcBH51b2/IwmUtOWVc7TSpsi4mdeQn8Xk0M6a4H9HtsU2tYJXGpmYeDtQE2myIyMF+/pd7ZqeEdE/Ktw8p3tPuBhM+sDbgI2mdmdzrnNM7TZQDbQp27bB3yH7BDPDuCHZTuTEowmio/pxyIhOlqimqsvIr40a+g7506a2QBwPfAV59wgsHuWNicACmx7HFhTzhMoVSbjGE0Un70DuRu01NMXER/y0tPHOXecyZk4ntt42a/aTicLL6t8pu641t8REX8K3B25I0UeoHKm7FIM6umLiP8ELvQnno8bLd7Tzw7vqKcvIv4TuNBPpjMANEWLn3pXvInjo0lSubYiIn4RuNBP5II8Gi5+6t25ufrHRjXEIyL+ErjQT6aztwbEZgj9Rbm7cvUELRHxm8CFfiI1e09f6++IiF8FLvSTE8M7VrRNfnhH0zZFxG8CF/oTY/oFHpWYN7nomnr6IuIvgQv9ZG54Z6Yx/Y6WKOGQaSkGEfGd4IV+7kLuTGP6oZCxSA9IFxEfCmDo53r6MwzvQPaxiUcU+iLiM4EL/YSHC7mQfWyihndExG8CF/oTPf0ZhndAK22KiD8FL/Q9zNOH7Awerb8jIn4TvNDPX8idZUy/uz3GSCLNaG5VThERPwhc6Jcypg9w5JSGeETEPwIX+hN35IZmPvWe9mzoDw2PVbwmEZFqCWToR0JGKDRzT78n19MfUk9fRHwkcKGfSGVmvYgLZ/b0dTFXRPwjcKGfTLtZx/MBFsVjmMGRUwp9EfGPwIV+Ip2Z9W5cyE7p7GyNqacvIr4SuNBPehzegey4vnr6IuInwQv9tPfQ725XT19E/CWAoe88De9Arqev0BcRHwlc6CdK6On3tDcxdGoc51yFqxIRqY7AhX4ynSHmYfYOZO/KHUtmGB7XUgwi4g+BDP1SevqA1tUXEd8IXuinnPcLuRN35WpcX0T8IXChn0hnZl1hM2+yp6/QFxF/CFzolzqmD+rpi4h/BDL0vQ7vLIrHCJl6+iLiHwEMfe9j+uGQ0dXWpJ6+iPiGp/Qzs7vNbIeZbS6lTbH9zOwuM9s497Lnzusqm3ndCn0R8ZFZ08/MbgbCzrkrgJVmttpLm2L7mdnVQK9z7mdlPROPsguueRvTh+zFXA3viIhfeOnyDgBbc68fAK7y2GbaNjOLAn8P7Dez9xZ6MzO71cx2mtnOoaEhD+WVppQxfYDutph6+iLiG17SLw4czL0+Bizx2KbQtg8DzwBfAdab2aenHsg5t8U5t845t66np8freXhWyiqbkO/pJ7QUg4j4gpf0GwZacq/biuxTqE2hbZcDW5xzg8APgGvnVvbclXIhF7KLriXSGU6e1lIMItL4vKTfLiaHdNYC+z22KbRtH7Ayt20d8HKJ9c6Lc87zQ1Ty9NhEEfGTiIc29wEPm1kfcBOwyczudM5tnqHNBsAV2JYBvm1mm4Ao8G/KdyqzS2WyQzReb86CMx+QPs6qxW0VqUtEpFpmDX3n3EkzGwCuB76SG5rZPUubEwCFtgH/tmzVlyiZzgCUPKYPcPjUWEVqEhGpJi89fZxzx5mcieO5jZf9qimZyvb0Swn9voXZyxIHjp+uSE0iItUUqDtyE/mefglj+vGmCF3xGAeOj1aqLBGRqglU6OeHd0oZ0wdYtqiVV4+ppy8ijS+QoV/K8A7A8s4WXlVPX0R8QKHvwfJFrbz2xmnSGd2gJSKNLVChP56aa0+/lWTaMXhSM3hEpLEFKvST6dw8/RIWXANYvig7g+fVYxriEZHGFrDQn3tPHxT6ItL4ghX6cxzeWbqwGTN4VXP1RaTBBSr0E3Ps6TdFwvQuaOaAevoi0uACFfr5Mf2mEm7Oylve2appmyLS8AIW+nPr6QMsW9SiG7REpOEFNPRLm70D2Z7+oVNjjKfS5S5LRKRqAhX6iTleyIXsDVrOwUFdzBWRBhao0J+cpz+XMf3cXH2Fvog0sICF/vx6+oBW2xSRhhbQ0C99TH/JgmaiYdPFXBFpaIEK/bnO0wcIh4xzFmq1TRFpbMEK/XlcyIXsEM8rRxX6ItK4AhX6yXSGcMgIh0of3gFY0R1n/5ERnNMSyyLSmAIW+m5O4/l553XFOTWe4thIooxViYhUT6BCP5HKzHloB2BFd3YGz/6jI+UqSUSkqgIV+sl0htg8Qr+/Kw7A/iMa1xeRxhS40J9PT39ZZyshU09fRBpXwELfzelu3LxYJMSyzlZeOqLQF5HGFKjQT6Qz87qQC9DfHedlTdsUkQYVqNBPzvNCLkB/V6umbYpIwwpW6Kcz8xregezF3FPjKY5q2qaINKCAhb6bf08/N23zZV3MFZEGFKjQL8uYfm7a5kuatikiDShQoT/fKZuQnbYZDhn7NYNHRBpQ4EJ/PjdnQXba5jkLWzRXX0QaUqBCf77LMOT1d8cV+iLSkDwloJndbWY7zGxzKW2K7WdmS8zsqbmXPTfJtCM6z9k7ACu6Wtl/ZFTTNkWk4cyagGZ2MxB2zl0BrDSz1V7azLLfV4GW8pyCd9me/vwu5EJ2tc3h8RRHhjVtU0Qai5du7wCwNff6AeAqj20K7mdm1wEjwGChNzOzW81sp5ntHBoa8lCed+UY0wc4f3EbAC8MDc/7WCIi1eQlAePAwdzrY8ASj22mbTOzGHA78Nlib+ac2+KcW+ecW9fT0+OhPO/KMXsH4E1LsqH//KFT8z6WiEg1eUnAYSaHYtqK7FOoTaFtnwXucs69MdeC52O+C67l9S5opr0pwvOH1dMXkcbiJQF3MTmksxbY77FNoW3vAG4zs+3Am83sW3Ooec4SZerpmxmrlrSxVz19EWkwEQ9t7gMeNrM+4CZgk5nd6ZzbPEObDYCbus059w/5Hcxsu3PuE+U6kdk453Jj+vO/kAuwenEb/7LncFmOJSJSLbN2e51zJ8lelH0UuNY5t3tK4Bdqc6LQtin7DJShfs/SGYdzlKWnD/CmJe0cGU7oebki0lA8JaBz7rhzbqtzruCMm2JtvOxXLcl0dk59OebpA6xarIu5ItJ4AnNHbiKdAcrb0wfYq4u5ItJAAhP6yVzol2tMf2lHM21NEfappy8iDSRwoV+unr6ZsWpxm6ZtikhDCUzoJ1LlDX3IzuDZe0ihLyKNIzChP9HTL9OFXMjP4BnnuGbwiEiDCEzoJ1LZ2TvlGtMHWJVfjkFDPCLSIAIT+uUe04fJGTzPH9bFXBFpDAr9eejLzeB5blChLyKNITChn5+nX44F1/LMjAt723n29ZNlO6aISCUFJvQn7sgtY08f4KKlC9jz+ik9RUtEGkJwQj+Vvzmr/KF/ajzFgeOny3pcEZFKCE7oT0zZLN/sHYCLlmYv5j6jIR4RaQCBCf1yr72Td0FvO2ZoXF9EGkJgQj8/pl/u4Z3WWIQVXXGFvog0hACFfmV6+pAd13/2dU3bFJH6F8DQL++YPmTH9V85NsqpsWTZjy0iUk6BCf2JBdfKOE8/76KlCwDYo5u0RKTOBSf005WZsgmToa9xfRGpd4EJ/WSqMjdnQfaBKh0tUYW+iNS94IR+OkPIIBwq/5i+mXHx0gU8o4u5IlLnAhP6iXSmIr38vIv7FrDn9ZOkcsNIIiL1KDChP5pIEW+KVOz4ly3rYDyV0ZO0RKSuBSb0R8bTtMbCFTv+2mULAdh94I2KvYeIyHwFJvSHx1O0VbCnf15XKx0tUXa/qtAXkfoVmNAfGa/s8I6ZcdmyDnYfOFGx9xARma/ghH4iXdHQB3jz8oXsPXSK04l0Rd9HRGSughP64ynamio3pg/Zcf10xvGH19TbF5H6FKjQj8cq29O/bHkHAE9rXF9E6lRgQn+4wmP6AIvbm+nraNa4vojUrUCEvnMudyG3ssM7AJctW8hvNW1TROpUIEJ/LJkh46h4Tx9g7fKFvHx0lOMjiYq/l4hIqQIR+iOJFEBF5+nnrc2P66u3LyJ1KBihP54N/UpfyAW4fHknsUiI3zx/pOLvJSJSKk+hb2Z3m9kOM9tcSpup28ysw8y2mdkDZvZjM4vN/xRmN5wP/Sr09FtiYd6+YhEP7R2q+HuJiJRq1tA3s5uBsHPuCmClma320qbIfn8KfM05dwMwCNxYzpMpZmQ8e7NUNYZ3AP7oTT3sOzzMgeOjVXk/ERGvvPT0B4CtudcPAFd5bDNtm3PuLufcr3LbeoDDUw9kZrea2U4z2zk0VJ7ecn54p7UKs3cABi7oAVBvX0TqjpfQjwMHc6+PAUs8tim6n5ldAXQ65x6deiDn3Bbn3Drn3Lqenh5PJzGb/PBOtXr65/e0cc7CFh56TqEvIvXFS+gPAy25121F9inUpuB+ZrYI+DrwsbmVXLrRRPXG9CG7+NofXdDDI/uOTDyQXUSkHngJ/V1MDumsBfZ7bDNtW+7C7Y+AzznnXp5byaUbzo/pV2H2Tt4fvamHkUSaXS8fr9p7iojMxksK3gc8bGZ9wE3AJjO70zm3eYY2GwBXYNvHgbcAf2lmfwl80zn3j+U7ncImpmxWaUwf4MpV3URCxkN7h7ji/K6qva+IyExm7ek7506SvSj7KHCtc273lMAv1OZEkW3fdM51OucGcn8qHviQDf2mSIhIBZ+RO1VbU4S39S/i/zx7qGrvKSIyG08p6Jw77pzb6pwbLKWNl/2qoRqLrRXyzjVL2Hd4mBeG9NxcEakPgbkjt5pDO3k3rOkF4Jd/qOnPPBGRCcEI/US6KkswTNW3sIW1yzr45e8V+iJSH4IR+hV+KPpM3nlJL7sPnOC1N07X5P1FRM4UmNCvxZg+wI25IZ4HNMQjInUgEKE/XMOe/sqeNlYvbuOfFfoiUgcCEfoj42laY9W/kJt34yW9PP7SMQ6fGqtZDSIiEJTQT9RueAfg5rcswwE/ePSVmtUgIgIBCP3883FrNbwDsKI7znUXLOaeR19mLJmuWR0iIr4P/Wo+H3cmH79qBUdHEvz06ddqWoeIBJvvQ39yWeXajekDXHF+Fxf2tvPtR17COVfTWkQkuHwf+iNVfFTiTMyMj121gj2Dp3hk39Ga1iIiweX70M/39FtrcEfuVH+8to/eBc3cef8zWmdfRGrC96E/mqju83Fn0hwN89/+5BL2DJ7im9tfqHU5IhJAvg/9WqylP5PrL17Ce9/cxzcefJ49gydrXY6IBIzvQ7/az8f14gsb17CgOcp//tFuxlOawiki1eP70K+XC7lnWhSP8cWbL+X3B0/y3+9/ttbliEiA+D70h+sw9AHeuaaXf3/1Cv7njpf5ydMHa12OiASE70N/JPdQ9HgN194p5jM3Xsjb+jv53D/9jsde1DROEak834f+aKL6z8f1KhoO8Y0PvIUlC5r5wLce428feoFMRjduiUjl1F8Sllktl1X2YsmCZn76qSu5cU0vX962h0/e86TW5xGRivF96NfyASpetTdH+cYHLmfzuy/il88M8uG7H+fE6WStyxIRH/J96A+Pp+s+9CG7TMMnrl7J32y6nKdePc77/nYHLw4N17osEfEZ34d+dlnl+ruIW8zGtX1896PrOXxqjPd8/Tf8aOerWqBNRMrG/6GfSNXFujuluHJVN9v+4zVctqyD//K/fsvtP/m9LvCKSFn4P/Tr/EJuMb0dzdzziQ3ces1KfvDoK/ynrU+TTGuRNhGZn8ZLwxKNjKfrZt2dUoVDxuffdREdLVH+xy+f48Dx0/zJ5ecwcEEPyzpba12eiDQgX4f+WDLNsdEEC1tjtS5lXm67dhVd8RjfeHAfm+/7PQCrF7dx7YWL+eO1fVxyTkeNKxSRRuHr0N/+3GESqQzXrO6pdSnztmn9ubz/bct5YWiE7c8dZvtzQ3znkZfY8usXuXp1N7ddu4q3r1iEmdW6VBGpY74O/ft/N8iieIwNKxfVupSyMDNWLW5j1eI2PnH1Sk6OJbnn0Ve4+zcvsmnLo1ywpJ1b1i+nb2ELw+MpwiHj3EWt9HfF6Yw39m87IlIevg39sWSa//vsId775r66XIKhHBY0R/nzgfP56JX93PfUQf7h8Ve442fPFGx70dIF/KsLF7NxbR8X9LZXuVIRqRe+Df3tzw0xmkjzrkuX1rqUimuOhtm0/lw2rT+XfYdPMZbMEG+KkEhleOXYKHsPneKhvUN886EX+MaD+7h6dTcf3HAea/oW0NfRQiikISGRoPBt6P/id6/T2RrlipVdtS6lqlYtPrsXf0FvO9dfvITbrl3FsZEE9z7xCt/7f/v5D9/fBUBTJERTJIRz0N4cYfWSdt60pI1zFrbQ29FCLGITK5WuXtLGyu42YpHJ35wyGceR4XFeOzHG4InTLGyNcdmyjoa7N0IkKDz9n2lmdwMXA/c75+702sbrtnLLD+1sXOvfoZ25WBSP8cmBVXziqpU8+cpxXhwaYf/RERKpDGZwfCTB3kPD7HjxaNEHt0dCxsLWGO3NEZLpDIdOjpFMn33jWDhknLeolVgkRMiMcMiyv004x3gqQyrj6GiJsigeIxYOkcpkCJnR0RJlYWuMztYona0xFrZG6Yxn/97WFCXeFCbjsiunjoynz/5vIo1zjgUtUdqaImR/eTHy17UzGcfweLZ9JneHczRsNEfDtETDtMSy/23O/YmEjLFUmkQqQ3M0TLwpQiwcYup1csudb7kuoKfSGZJphxmEzAjZ9OOnM46xZPY8wiGb+BqHzUr+rc05h3OQ/w4aYEZJ55M/BjPs65wjmXYk0plcnRDO1T2fr13+uOlM9muW/3qESjyHcnLOkXHZ75PD5T43k7VkMo5kJvt9zn+/o2GjvTlKOGQ4l/06ATRFyj/dfNbQN7ObgbBz7goz+7aZrXbOPT9bG+BSL9umHqscHto7xEhAhnbmIhYJsWFlFxuK/BaUyTiOjSZ4/Y0x0s4Rj4VJph3PHz7F3kOnODaSzF4oNli6sIW+jmaWdrTQ29HM0KlxnnzlOC8MDZNKOzIu+z9k2mUDpSkSIhwyTpxO8uqxUVIZRyRkpDKOE6eTvDGamPZDpBGYMRG6c4kaRzYk0jPceZ0PspnaQD74oFgljmwoZc4I62LM8j8IbOIHAmSPnXGO9CzHyO/voGi70BlhXUpOOweJdGbG9w9N/PC03A+kGWqd5TtXbF/nIO0cmUzxr0dTJIQj+0N9pm9fUyQ0cU6fHDifz9x44Yw1zYXNtq6Lmf0N8M/OuV+Y2SagxTn3ndnaAJd72VbgWLcCt+b+egHw3PxP8yzdwJEyH7McVFdpVFdpVFfp6rU2L3Wd55wrOFfdy/BOHMg/z+8Y8BaPbbxuO4tzbguwxUNdc2JmO51z6yp1/LlSXaVRXaVRXaWr19rmW5eXAe9hsr10gLYi+xRq43WbiIhUiZfQ3QVclXu9FtjvsY3XbSIiUiVehnfuAx42sz7gJmCTmd3pnNs8Q5sNZK/deNlWbRUbOpon1VUa1VUa1VW6eq1tXnXNeiEXwMw6geuBXzvnBr228bpNRESqw1Poi4iIP+hCqohIgAQq9M3sbjPbYWabZ29dkffvMLNtZvaAmf3YzGKFaqpVnWa2xMyeKlZDDeu6y8w21ktdZtZpZr8ws51m9nf1UFfue/fwTO9dixrPrKvQ578e6pqy7amZaqhRXROf/3LUFZjQP/OuYWBl7g7havtT4GvOuRuAQWDT1JpqXOdXgZZCNdSqLjO7Guh1zv2sjur6EHBPbq50u5l9ppZ15a6TfY/sfTAFP+u1+NpNrYvpn/8b66SuvK+Sm1JeL3Wd+fkvV12BCX1gANiae/0Ak1NHq8Y5d5dz7le5v/YAHyxQ00CBbRVnZtcBI2T/ZyxUQ9XrMrMo8PfAfjN7b73UBRwFLjGzhcByYEWN60oD7wdO5v5e6L29bqtYXQU+/4froS6Y9vmnHuoq8PkvS11BCv2pdwMvqVUhZnYF0Am8WqCmqteZ+zX7duCzuU2FaqjF1+/DwDPAV4D1wG11UtdvgPOAvwCeBWK1rMs5d9I5d+KMTV6/fxWtsUBdwOTn3zn3aD3UVeDzTz3UxZTPv5l9uhx1BSn06+JuYDNbBHwd+FiRmmpR52eBu5xzb+T+Xi91XQ5syU3t/QHw6zqp6wvAnznn/grYA3ygTurKq9s75Kd8/kuptZKmfv7rpa6pn/9ry1FXkEK/5ncD53oUPwI+55x7uUhNtfplKcEAAADfSURBVKjzHcBtZrYdeDOwsU7q2geszL1eB/TXSV2dwKVmFgbeDny5TurKq8s75At8/kuptZLO+vyb2bfqpK6pn//yZEZ2LWz//wEWALuBr5H9lbyjBjX8OXAc2J778++m1lTrOnN1TauhFnUB7WRD4tfADrJDKvVQ13rgD2R7WL+qo6/X9mKf9VrWeEZdUz//76+HuubzNazw12vq5/+cctQVqJuzrA7vBi5UUz3Uqboauy6v9ehrF7y6AhX6IiJBF6QxfRGRwFPoi4gEiEJfRCRAFPoiIgGi0BcRCZD/DyIA7LHLx35bAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import re\n",
    "train_df['punc_num'] = train_df['text'].apply(lambda x: len(re.findall(\"(3750|900|648)\",x)))\n",
    "sns.kdeplot(train_df['punc_num'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    10000.000000\n",
       "mean        79.587100\n",
       "std         86.911722\n",
       "min          0.000000\n",
       "25%         28.000000\n",
       "50%         56.000000\n",
       "75%        101.250000\n",
       "max       1542.000000\n",
       "Name: punc_num, dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['punc_num'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "label\n",
       "0     3750 648 900 3370 4464\n",
       "1     3750 648 3370 900 4464\n",
       "2     3750 648 900 7399 6122\n",
       "3     3750 648 900 6122 4939\n",
       "4     3750 648 900 4411 7399\n",
       "5     3750 648 900 6122 5598\n",
       "6     3750 648 900 6248 2555\n",
       "7     3750 648 900 3370 5296\n",
       "8      3750 648 900 6122 913\n",
       "9     3750 648 900 2465 6122\n",
       "10    3750 648 900 3370 2465\n",
       "11    3750 648 900 4939 6122\n",
       "12    3750 4464 3370 900 648\n",
       "13    3750 648 900 4939 6122\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def fun(x):   \n",
    "    all_lines = ' '.join(list(x))\n",
    "    word_count = Counter(all_lines.split(\" \"))\n",
    "    word_count = sorted(word_count.items(), key=lambda d:d[1], reverse = True)  # ascending order\n",
    "    return \" \".join([item[0] for item in word_count[:5]])\n",
    "train_df.groupby([\"label\"])['text'].apply(fun)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**关于Datawhale：**\n",
    "\n",
    "> Datawhale是一个专注于数据科学与AI领域的开源组织，汇集了众多领域院校和知名企业的优秀学习者，聚合了一群有开源精神和探索精神的团队成员。Datawhale 以“for the learner，和学习者一起成长”为愿景，鼓励真实地展现自我、开放包容、互信互助、敢于试错和勇于担当。同时 Datawhale 用开源的理念去探索开源内容、开源学习和开源方案，赋能人才培养，助力人才成长，建立起人与人，人与知识，人与企业和人与未来的联结。\n",
    "\n",
    "本次新闻文本分类学习，专题知识将在天池分享，详情可关注Datawhale：\n",
    "\n",
    " ![](http://jupter-oss.oss-cn-hangzhou.aliyuncs.com/public/files/image/1095279172547/1584432602983_kAxAvgQpG2.jpg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}