Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix task hints from strides to dilations #154

Open
wants to merge 1 commit into
base: 2024
Choose a base branch
from
Open

Fix task hints from strides to dilations #154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions week03_lm/homework_pytorch.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,8 +192,8 @@
" Note: fixed window LM is effectively performing a convolution over a sequence of words.\n",
" This convolution only looks on current and previous words.\n",
" Such convolution can be represented as a sequence of 2 operations:\n",
" - pad input vectors by {strides * (filter_size - 1)} zero vectors on the \"left\", do not pad right\n",
" - perform regular convolution with {filter_size} and {strides}\n",
" - pad input vectors by {dilation * (filter_size - 1)} zero vectors on the \"left\", do not pad right\n",
" - perform regular convolution with {filter_size} and {dilations}\n",
" \n",
" - If you're absolutely lost, here's a hint: use nn.ZeroPad2d((NUM_LEADING_ZEROS, 0, 0, 0))\n",
" followed by a nn.Conv1d(..., padding=0). And yes, its okay that padding is technically \"2d\".\n",
Expand Down Expand Up @@ -789,7 +789,7 @@
"So you've learned the building blocks of neural language models, you can now build the ultimate monster: \n",
"* Make it char-level, word level or maybe use sub-word units like [bpe](https://github.com/rsennrich/subword-nmt);\n",
"* Combine convolutions, recurrent cells, pre-trained embeddings and all the black magic deep learning has to offer;\n",
" * Use strides to get larger window size quickly. Here's a [scheme](https://storage.googleapis.com/deepmind-live-cms/documents/BlogPost-Fig2-Anim-160908-r01.gif) from google wavenet.\n",
" * Use dilations to get larger window size quickly. Here's a [scheme](https://storage.googleapis.com/deepmind-live-cms/documents/BlogPost-Fig2-Anim-160908-r01.gif) from google wavenet.\n",
"* Train on large data. Like... really large. Try [1 Billion Words](http://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz) benchmark;\n",
"* Use training schedules to speed up training. Start with small length and increase over time; Take a look at [one cycle](https://medium.com/@nachiket.tanksale/finding-good-learning-rate-and-the-one-cycle-policy-7159fe1db5d6) for learning rate;\n",
"\n",
Expand Down
6 changes: 3 additions & 3 deletions week03_lm/homework_tf.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,8 +193,8 @@
" Note: fixed window LM is effectively performing a convolution over a sequence of words.\n",
" This convolution only looks on current and previous words.\n",
" Such convolution can be represented as a sequence of 2 operations:\n",
" - pad input vectors by {strides * (filter_size - 1)} zero vectors on the \"left\", do not pad right\n",
" - perform regular convolution with {filter_size} and {strides}\n",
" - pad input vectors by dilation * (filter_size - 1)} zero vectors on the \"left\", do not pad right\n",
" - perform regular convolution with {filter_size} and {dilations}\n",
" - If you're absolutely lost, here's a hint: use ZeroPadding1D and Conv1D from keras.layers\n",
" You can stack several convolutions at once\n",
" \"\"\"\n",
Expand Down Expand Up @@ -831,7 +831,7 @@
"So you've learned the building blocks of neural language models, you can now build the ultimate monster: \n",
"* Make it char-level, word level or maybe use sub-word units like [bpe](https://github.com/rsennrich/subword-nmt);\n",
"* Combine convolutions, recurrent cells, pre-trained embeddings and all the black magic deep learning has to offer;\n",
" * Use strides to get larger window size quickly. Here's a [scheme](https://storage.googleapis.com/deepmind-live-cms/documents/BlogPost-Fig2-Anim-160908-r01.gif) from google wavenet.\n",
" * Use dilations to get larger window size quickly. Here's a [scheme](https://storage.googleapis.com/deepmind-live-cms/documents/BlogPost-Fig2-Anim-160908-r01.gif) from google wavenet.\n",
"* Train on large data. Like... really large. Try [1 Billion Words](http://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz) benchmark;\n",
"* Use training schedules to speed up training. Start with small length and increase over time; Take a look at [one cycle](https://medium.com/@nachiket.tanksale/finding-good-learning-rate-and-the-one-cycle-policy-7159fe1db5d6) for learning rate;\n",
"\n",
Expand Down