From 0271b316b4aec02bdf4103cafc223136ce5cb748 Mon Sep 17 00:00:00 2001 From: Ivan Date: Tue, 8 Oct 2024 19:22:24 +0000 Subject: [PATCH] Fix task hints from strides to dilations --- week03_lm/homework_pytorch.ipynb | 6 +++--- week03_lm/homework_tf.ipynb | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/week03_lm/homework_pytorch.ipynb b/week03_lm/homework_pytorch.ipynb index 5b97539d..51ba9b53 100644 --- a/week03_lm/homework_pytorch.ipynb +++ b/week03_lm/homework_pytorch.ipynb @@ -192,8 +192,8 @@ " Note: fixed window LM is effectively performing a convolution over a sequence of words.\n", " This convolution only looks on current and previous words.\n", " Such convolution can be represented as a sequence of 2 operations:\n", - " - pad input vectors by {strides * (filter_size - 1)} zero vectors on the \"left\", do not pad right\n", - " - perform regular convolution with {filter_size} and {strides}\n", + " - pad input vectors by {dilation * (filter_size - 1)} zero vectors on the \"left\", do not pad right\n", + " - perform regular convolution with {filter_size} and {dilations}\n", " \n", " - If you're absolutely lost, here's a hint: use nn.ZeroPad2d((NUM_LEADING_ZEROS, 0, 0, 0))\n", " followed by a nn.Conv1d(..., padding=0). And yes, its okay that padding is technically \"2d\".\n", @@ -789,7 +789,7 @@ "So you've learned the building blocks of neural language models, you can now build the ultimate monster: \n", "* Make it char-level, word level or maybe use sub-word units like [bpe](https://github.com/rsennrich/subword-nmt);\n", "* Combine convolutions, recurrent cells, pre-trained embeddings and all the black magic deep learning has to offer;\n", - " * Use strides to get larger window size quickly. Here's a [scheme](https://storage.googleapis.com/deepmind-live-cms/documents/BlogPost-Fig2-Anim-160908-r01.gif) from google wavenet.\n", + " * Use dilations to get larger window size quickly. Here's a [scheme](https://storage.googleapis.com/deepmind-live-cms/documents/BlogPost-Fig2-Anim-160908-r01.gif) from google wavenet.\n", "* Train on large data. Like... really large. Try [1 Billion Words](http://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz) benchmark;\n", "* Use training schedules to speed up training. Start with small length and increase over time; Take a look at [one cycle](https://medium.com/@nachiket.tanksale/finding-good-learning-rate-and-the-one-cycle-policy-7159fe1db5d6) for learning rate;\n", "\n", diff --git a/week03_lm/homework_tf.ipynb b/week03_lm/homework_tf.ipynb index 0541ec6b..d4b5b249 100644 --- a/week03_lm/homework_tf.ipynb +++ b/week03_lm/homework_tf.ipynb @@ -193,8 +193,8 @@ " Note: fixed window LM is effectively performing a convolution over a sequence of words.\n", " This convolution only looks on current and previous words.\n", " Such convolution can be represented as a sequence of 2 operations:\n", - " - pad input vectors by {strides * (filter_size - 1)} zero vectors on the \"left\", do not pad right\n", - " - perform regular convolution with {filter_size} and {strides}\n", + " - pad input vectors by dilation * (filter_size - 1)} zero vectors on the \"left\", do not pad right\n", + " - perform regular convolution with {filter_size} and {dilations}\n", " - If you're absolutely lost, here's a hint: use ZeroPadding1D and Conv1D from keras.layers\n", " You can stack several convolutions at once\n", " \"\"\"\n", @@ -831,7 +831,7 @@ "So you've learned the building blocks of neural language models, you can now build the ultimate monster: \n", "* Make it char-level, word level or maybe use sub-word units like [bpe](https://github.com/rsennrich/subword-nmt);\n", "* Combine convolutions, recurrent cells, pre-trained embeddings and all the black magic deep learning has to offer;\n", - " * Use strides to get larger window size quickly. Here's a [scheme](https://storage.googleapis.com/deepmind-live-cms/documents/BlogPost-Fig2-Anim-160908-r01.gif) from google wavenet.\n", + " * Use dilations to get larger window size quickly. Here's a [scheme](https://storage.googleapis.com/deepmind-live-cms/documents/BlogPost-Fig2-Anim-160908-r01.gif) from google wavenet.\n", "* Train on large data. Like... really large. Try [1 Billion Words](http://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz) benchmark;\n", "* Use training schedules to speed up training. Start with small length and increase over time; Take a look at [one cycle](https://medium.com/@nachiket.tanksale/finding-good-learning-rate-and-the-one-cycle-policy-7159fe1db5d6) for learning rate;\n", "\n",