1.4_snapshot_explain.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "487b4777-9981-4de5-a440-916ee4af0513",
   "metadata": {},
   "source": [
    "# Snapshot Explain\n",
    "\n",
    "This notebook shows how to run a Snapshot Explain operation with the minimal steps and a simple query.\n",
    "\n",
    "In this notebook...\n",
    "* [Dependencies and Initialisation](#dependencies-and-initialisation)\n",
    "* [The Where Statement](#the-where-statement)\n",
    "* [Running the Explain Operation](#running-the-explain-operation)\n",
    "* [Next Steps](#next-steps)\n",
    "\n",
    "**_NOTE_**: This notebook was tested using the [`factiva-analytics`](https://factiva-analytics-python.readthedocs.io/) Python package version **0.3.13** (Sep 2024) and [Factiva Analytics v3.0](https://developer.dowjones.com/site/docs/factiva_apis/factiva_analytics_apis/factiva_snapshots_api/snapshots_3_0/index.gsp#) endpoints."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd55c3bc-ffc6-4634-923f-aa8aee26d7d0",
   "metadata": {},
   "source": [
    "## Dependencies and Initialisation\n",
    "Import statements and environment initialisation using the package `dotenv`. More details in the [Configuration notebook](0.2_configuration.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7535aef1-faff-4eb1-b798-4d453468d498",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using the Factiva Analytics Python Package version 0.3.13\n"
     ]
    }
   ],
   "source": [
    "import factiva.analytics as fa\n",
    "from factiva.analytics import SnapshotExplain\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()\n",
    "print(f\"Using the Factiva Analytics Python Package version {fa.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0273c8a9-6bb6-447b-abaf-b81a87f6511d",
   "metadata": {},
   "source": [
    "## The Where Statement\n",
    "\n",
    "This notebook uses a simple query for illustration purposes. For more tips about queries, or guidance on how to build complex or large queries, checkout the [query reference](2.1_complex_large_queries.ipynb) notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1816e3a2-9258-4d4d-a737-a1fd1e45fd31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Industry i3432 is for Batteries\n",
    "where_statement = (\n",
    "    r\" publication_datetime >= '2016-01-01 00:00:00' \"\n",
    "    r\" AND LOWER(language_code) IN ('en', 'de', 'fr') \"\n",
    "    r\" AND REGEXP_CONTAINS(industry_codes, r'(?i)(^|,)(i3432)($|,)') \"\n",
    ")\n",
    "\n",
    "expl = SnapshotExplain(query=where_statement)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8bf7ac0-0a75-411c-9809-781bdca9aa9b",
   "metadata": {},
   "source": [
    "## Running the Explain Operation\n",
    "\n",
    "[Product Documentation - Snapshot Explain](https://developer.dowjones.com/site/docs/factiva_apis/factiva_analytics_apis/factiva_snapshots_api/snapshots_3_0/index.gsp#snapshotexplains-13)\n",
    "\n",
    "This operation returns the number of documents matching the provided query in the Factiva Analytics archive.\n",
    "\n",
    "The goal of this operation is to have a rough idea of the document volume. When used iteratively, helps deciding on the used criteria to add/delete/modify the criteria to verify the impact on the matching items.\n",
    "\n",
    "\n",
    "The `<SnapshotExplain>.process_job()` function directly returns this value. If a more manual process is required (run submit, monitor or get job results as separate operations), please see the [detailed package documentation](https://factiva-analytics-python.readthedocs.io/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a495fac7-5096-43d6-9d0b-4d083cf02409",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 94.8 ms, sys: 21.9 ms, total: 117 ms\n",
      "Wall time: 59.4 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "expl.process_job()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c2cc68eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Explain operation ID: 72e8e01e-fcce-4b23-8a2d-2fbb65e84fc2\n",
      "Document volume estimate: 321078\n"
     ]
    }
   ],
   "source": [
    "print(f'Explain operation ID: {expl.job_response.job_id}')\n",
    "print(f'Document volume estimate: {expl.job_response.volume_estimate}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06ed064c",
   "metadata": {},
   "source": [
    "## Getting Explain Samples\n",
    "\n",
    "[Product documentation - Snapshot Explain Samples](https://developer.dowjones.com/site/docs/factiva_apis/factiva_analytics_apis/factiva_snapshots_api/snapshots_3_0/index.gsp#snapshotsamples-15)\n",
    "\n",
    "As an extension of the Explain operation, it is possible to request a set of random article metadata samples matching the Explain criteria. The main requirement in this case is just using the previously obtained Explain Job ID. It uses the ID from the `last_explain_job` instance within the `Snapshot` instance.\n",
    "\n",
    "The operation `get_samples(num_samples=10)` runs the API operation and stores the result into the same `<SnapshotExplain>`'s `samples` property. The response is essentially metadata content from a random selection of items. It accepts the parameter `num_samples` which can be an integer between `1` and `100`.\n",
    "\n",
    "**_NOTE:_: It's important to run `<SnapshotExplain>.get_samples()` within the next 30 minutes of an explain. Otherwise no samples will be retrieved as server cache is flushed frequently.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3a7b1861",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "expl.get_samples(num_samples=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b7f9ca3",
   "metadata": {},
   "source": [
    "By checking the value of `<SnapshotExplain>.samples.data.columns` it's possible to see the list of columns that can be used for a quick analysis. The property `data` is a `Pandas.DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "28b4bc04",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['an', 'company_codes', 'company_codes_about', 'company_codes_occur',\n",
       "       'industry_codes', 'ingestion_datetime', 'modification_datetime',\n",
       "       'publication_datetime', 'publisher_name', 'region_codes',\n",
       "       'region_of_origin', 'source_code', 'source_name', 'subject_codes',\n",
       "       'title', 'word_count', 'newswires_codes', 'restrictor_codes'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "expl.samples.data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "122522ca",
   "metadata": {},
   "source": [
    "Displaying a subset of the content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f67001a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>an</th>\n",
       "      <th>source_name</th>\n",
       "      <th>title</th>\n",
       "      <th>word_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B000000020180317ee3j0005l</td>\n",
       "      <td>Barron's</td>\n",
       "      <td>A Cheaper Electric-Car Play Than Tesla</td>\n",
       "      <td>863</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B000000020210130eh2100001</td>\n",
       "      <td>Barron's</td>\n",
       "      <td>An EV Play Driven Solely By Promise --- Quantu...</td>\n",
       "      <td>1141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B000000020210925eh9r000b5</td>\n",
       "      <td>Barron's</td>\n",
       "      <td>EV Battery Recycler Starts With a Jolt --- New...</td>\n",
       "      <td>998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B000000020210911eh9d0002t</td>\n",
       "      <td>Barron's</td>\n",
       "      <td>The Great EV Battery Race</td>\n",
       "      <td>771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B000000020170225ed2r0008f</td>\n",
       "      <td>Barron's</td>\n",
       "      <td>This Week: Preview</td>\n",
       "      <td>394</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>B000000020190928ef9u000bl</td>\n",
       "      <td>Barron's</td>\n",
       "      <td>Electric-Car Battery Dilemma</td>\n",
       "      <td>618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>B000000020211030ehb100003</td>\n",
       "      <td>Barron's</td>\n",
       "      <td>Building A Better Battery --- Auto makers and ...</td>\n",
       "      <td>2571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>B000000020170429ed510008p</td>\n",
       "      <td>Barron's</td>\n",
       "      <td>Cleaning Up for a Buyer A Closer Look --- Edge...</td>\n",
       "      <td>999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>B000000020201225egcs000ul</td>\n",
       "      <td>Barron's</td>\n",
       "      <td>Epic Rise of a Battery Maker</td>\n",
       "      <td>845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>B000000020160507ec590000l</td>\n",
       "      <td>Barron's</td>\n",
       "      <td>Sizing Up Small-Caps: Edgewell Could Fetch a 4...</td>\n",
       "      <td>880</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          an source_name  \\\n",
       "0  B000000020180317ee3j0005l    Barron's   \n",
       "1  B000000020210130eh2100001    Barron's   \n",
       "2  B000000020210925eh9r000b5    Barron's   \n",
       "3  B000000020210911eh9d0002t    Barron's   \n",
       "4  B000000020170225ed2r0008f    Barron's   \n",
       "5  B000000020190928ef9u000bl    Barron's   \n",
       "6  B000000020211030ehb100003    Barron's   \n",
       "7  B000000020170429ed510008p    Barron's   \n",
       "8  B000000020201225egcs000ul    Barron's   \n",
       "9  B000000020160507ec590000l    Barron's   \n",
       "\n",
       "                                               title  word_count  \n",
       "0             A Cheaper Electric-Car Play Than Tesla         863  \n",
       "1  An EV Play Driven Solely By Promise --- Quantu...        1141  \n",
       "2  EV Battery Recycler Starts With a Jolt --- New...         998  \n",
       "3                          The Great EV Battery Race         771  \n",
       "4                                 This Week: Preview         394  \n",
       "5                       Electric-Car Battery Dilemma         618  \n",
       "6  Building A Better Battery --- Auto makers and ...        2571  \n",
       "7  Cleaning Up for a Buyer A Closer Look --- Edge...         999  \n",
       "8                       Epic Rise of a Battery Maker         845  \n",
       "9  Sizing Up Small-Caps: Edgewell Could Fetch a 4...         880  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "expl.samples.data[['an', 'source_name', 'title', 'word_count']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a6474a3",
   "metadata": {},
   "source": [
    "# Next Steps\n",
    "\n",
    "* Run an [analytics](1.5_snapshot_analytics.ipynb) to get a detailed time-series dataset of the estimates.\n",
    "* Run an [extraction](1.6_snapshot_extraction.ipynb) and download the matching content.\n",
    "* Fine-tune the query by adding/modifying the query criteria (where_statement) according to the [query reference](2.1_complex_large_queries.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fadev",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}