Skip to content

Commit 60544a3

Browse files
rguthrie3rguthrie3
committed
BiLSTM CRF commenting and more examples.
1 parent e548bd6 commit 60544a3

File tree

2 files changed

+177
-29
lines changed

2 files changed

+177
-29
lines changed

‎Deep Learning for Natural Language Processing with Pytorch.ipynb

Lines changed: 175 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -13,18 +13,18 @@
1313
},
1414
{
1515
"cell_type": "code",
16-
"execution_count": 2,
16+
"execution_count": 1,
1717
"metadata": {
1818
"collapsed": false
1919
},
2020
"outputs": [
2121
{
2222
"data": {
2323
"text/plain": [
24-
"<torch._C.Generator at 0x7f3085495af8>"
24+
"<torch._C.Generator at 0x10ee09738>"
2525
]
2626
},
27-
"execution_count": 2,
27+
"execution_count": 1,
2828
"metadata": {},
2929
"output_type": "execute_result"
3030
}
@@ -591,6 +591,38 @@
591591
"$$ f(x) = Ax + b $$ for a matrix $A$ and vectors $x, b$. The parameters to be learned here are $A$ and $b$. Often, $b$ is refered to as the *bias* term."
592592
]
593593
},
594+
{
595+
"cell_type": "markdown",
596+
"metadata": {},
597+
"source": [
598+
"Pytorch and most other deep learning frameworks do things a little differently than traditional linear algebra. It maps the rows of the input instead of the columns. That is, the $i$'th row of the output below is the mapping of the $i$'th row of the input under $A$, plus the bias term. Look at the example below."
599+
]
600+
},
601+
{
602+
"cell_type": "code",
603+
"execution_count": 4,
604+
"metadata": {
605+
"collapsed": false
606+
},
607+
"outputs": [
608+
{
609+
"name": "stdout",
610+
"output_type": "stream",
611+
"text": [
612+
"Variable containing:\n",
613+
"-0.0313 0.3256 0.5485\n",
614+
"-0.2189 -0.0064 -0.0617\n",
615+
"[torch.FloatTensor of size 2x3]\n",
616+
"\n"
617+
]
618+
}
619+
],
620+
"source": [
621+
"lin = nn.Linear(5, 3) # maps from R^5 to R^3, parameters A, b\n",
622+
"data = autograd.Variable( torch.randn(2, 5) ) # data is 2x5. A maps from 5 to 3... can we map \"data\" under A?\n",
623+
"print lin(data) # yes"
624+
]
625+
},
594626
{
595627
"cell_type": "markdown",
596628
"metadata": {},
@@ -613,6 +645,39 @@
613645
"A quick note: although you may have learned some neural networks in your intro to AI class where $\\sigma(x)$ was the default non-linearity, typically people shy away from it in practice. This is because the gradient *vanishes* very quickly as the absolute value of the argument grows. Small gradients means it is hard to learn. Most people default to tanh or ReLU."
614646
]
615647
},
648+
{
649+
"cell_type": "code",
650+
"execution_count": 5,
651+
"metadata": {
652+
"collapsed": false
653+
},
654+
"outputs": [
655+
{
656+
"name": "stdout",
657+
"output_type": "stream",
658+
"text": [
659+
"Variable containing:\n",
660+
"-0.2067 1.0672\n",
661+
" 0.1732 -0.6873\n",
662+
"[torch.FloatTensor of size 2x2]\n",
663+
"\n",
664+
"Variable containing:\n",
665+
" 0.0000 1.0672\n",
666+
" 0.1732 0.0000\n",
667+
"[torch.FloatTensor of size 2x2]\n",
668+
"\n"
669+
]
670+
}
671+
],
672+
"source": [
673+
"# In pytorch, most non-linearities are in torch.functional (we have it imported as F)\n",
674+
"# Note that non-linearites typically don't have parameters like affine maps do.\n",
675+
"# That is, they don't have weights that are updated during training.\n",
676+
"data = autograd.Variable( torch.randn(2, 2) )\n",
677+
"print data\n",
678+
"print F.relu(data)"
679+
]
680+
},
616681
{
617682
"cell_type": "markdown",
618683
"metadata": {},
@@ -625,6 +690,57 @@
625690
"You could also think of it as just applying an element-wise exponentiation operator to the input (to make everything non-negative) and then dividing by the normalization constant."
626691
]
627692
},
693+
{
694+
"cell_type": "code",
695+
"execution_count": 7,
696+
"metadata": {
697+
"collapsed": false
698+
},
699+
"outputs": [
700+
{
701+
"name": "stdout",
702+
"output_type": "stream",
703+
"text": [
704+
"Variable containing:\n",
705+
"-0.2443\n",
706+
"-0.5850\n",
707+
" 2.0812\n",
708+
"-0.1186\n",
709+
" 0.4903\n",
710+
"[torch.FloatTensor of size 5]\n",
711+
"\n",
712+
"Variable containing:\n",
713+
" 0.0660\n",
714+
" 0.0469\n",
715+
" 0.6748\n",
716+
" 0.0748\n",
717+
" 0.1375\n",
718+
"[torch.FloatTensor of size 5]\n",
719+
"\n",
720+
"Variable containing:\n",
721+
" 1\n",
722+
"[torch.FloatTensor of size 1]\n",
723+
"\n",
724+
"Variable containing:\n",
725+
"-2.7188\n",
726+
"-3.0595\n",
727+
"-0.3933\n",
728+
"-2.5931\n",
729+
"-1.9841\n",
730+
"[torch.FloatTensor of size 5]\n",
731+
"\n"
732+
]
733+
}
734+
],
735+
"source": [
736+
"# Softmax is also in torch.functional\n",
737+
"data = autograd.Variable( torch.randn(5) )\n",
738+
"print data\n",
739+
"print F.softmax(data)\n",
740+
"print F.softmax(data).sum() # Sums to 1 because it is a distribution!\n",
741+
"print F.log_softmax(data) # theres also log_softmax"
742+
]
743+
},
628744
{
629745
"cell_type": "markdown",
630746
"metadata": {},
@@ -1699,8 +1815,19 @@
16991815
"cell_type": "markdown",
17001816
"metadata": {},
17011817
"source": [
1702-
"### Example: An LSTM Language Model\n",
1703-
"TODO"
1818+
"### Exercise: Augmenting the LSTM part-of-speech tagger with character-level features\n",
1819+
"In the example above, each word had an embedding, which served as the inputs to our sequence model.\n",
1820+
"Let's augment the word embeddings with a representation derived from the characters of the word.\n",
1821+
"We expect that this should help significantly, since character-level information like affixes have\n",
1822+
"a large bearing on part-of-speech. For example, words with the affix *-ly* are almost always tagged as adverbs in English.\n",
1823+
"\n",
1824+
"Do do this, let $c_w$ be the character-level representation of word $w$. Let $x_w$ be the word embedding as before.\n",
1825+
"Then the input to our sequence model is the concatenation of $x_w$ and $c_w$. So if $x_w$ has dimension 5, and $c_w$ dimension 3, then our LSTM should accept an input of dimension 8.\n",
1826+
"\n",
1827+
"To get the character level representation, do an LSTM over the characters of a word, and let $c_w$ be the final hidden state of this LSTM.\n",
1828+
"Hints:\n",
1829+
"* There are going to be two LSTM's in your new model. The original one that outputs POS tag scores, and the new one that outputs a character-level representation of each word.\n",
1830+
"* To do a sequence model over characters, you will have to embed characters. The character embeddings will be the input to the character LSTM."
17041831
]
17051832
},
17061833
{
@@ -1745,12 +1872,25 @@
17451872
"source": [
17461873
"For this section, we will see a full, complicated example of a Bi-LSTM Conditional Random Field for named-entity recognition. Familiarity with CRF's is assumed. Although this name sounds scary, all the model is is a CRF but where an LSTM provides the features. This is an advanced model though, far more complicated than any earlier model in this tutorial. If you want to skip it, that is fine.\n",
17471874
"\n",
1748-
"TODO explain BiLSTM CRF Here"
1875+
"Recall that the CRF computes a conditional probability. Let $y$ be a tag sequence and $x$ an input sequence of words. Then we compute\n",
1876+
"$$ P(y|x) = \\frac{\\exp{(\\text{Score}(y)})}{\\sum_{y'} \\exp{(\\text{Score}(y')})} $$\n",
1877+
"\n",
1878+
"Where the score is determined by defining some log potentials $\\log \\psi_i(y)$ such that\n",
1879+
"$$ \\text{Score}(y) = \\sum_i \\log \\psi_i(y) $$\n",
1880+
"To make the partition function tractable, the potentials must look only at local features.\n",
1881+
"\n",
1882+
"In the Bi-LSTM CRF, we define two classes of potentials: emission and transition. The emission potential for the word at index $i$ comes from the hidden state of the Bi-LSTM at timestep $i$. The transition scores are stored in a $|T|x|T|$ matrix $\\textbf{P}$, where $T$ is the tag set.\n",
1883+
"\n",
1884+
"If the above discussion was too brief, you can check out [this](http://www.cs.columbia.edu/%7Emcollins/crf.pdf) write up from Michael Collins on CRFs.\n",
1885+
"\n",
1886+
"The example below implements the forward algorithm in log space to compute the partition function, and the viterbi algorithm to decode. Backpropagation will compute the gradients automatically for us. We don't have to do anything by hand.\n",
1887+
"\n",
1888+
"The implementation is not optimized. If you understand what is going on, you'll probably quickly see that iterating over the next tag in the forward algorithm could probably be done in one big operation. I wanted to code to be more readable."
17491889
]
17501890
},
17511891
{
17521892
"cell_type": "code",
1753-
"execution_count": 67,
1893+
"execution_count": 8,
17541894
"metadata": {
17551895
"collapsed": false
17561896
},
@@ -1759,26 +1899,27 @@
17591899
"data": {
17601900
"text/plain": [
17611901
"(Variable containing:\n",
1762-
" 1.8765\n",
1902+
" 3.1984\n",
17631903
" [torch.FloatTensor of size 1], [2, 1, 2])"
17641904
]
17651905
},
1766-
"execution_count": 67,
1906+
"execution_count": 8,
17671907
"metadata": {},
17681908
"output_type": "execute_result"
17691909
}
17701910
],
17711911
"source": [
1772-
"# Work in progress. Needs extensive commenting but it runs.\n",
1773-
"\n",
1774-
"\n",
1912+
"# Helper functions to make the code more readable.\n",
17751913
"def to_scalar(var):\n",
1914+
" # returns a python float\n",
17761915
" return var.view(-1).data.tolist()[0]\n",
17771916
"\n",
17781917
"def argmax(vec):\n",
1918+
" # return the argmax as a python int\n",
17791919
" _, idx = torch.max(vec, 1)\n",
17801920
" return to_scalar(idx)\n",
17811921
"\n",
1922+
"# Compute log sum exp in a numerically stable way for the forward algorithm\n",
17821923
"def log_sum_exp(vec):\n",
17831924
" max_score = vec[0][argmax(vec)]\n",
17841925
" max_score_broadcast = max_score.expand(vec.size()[1])\n",
@@ -1797,7 +1938,11 @@
17971938
" \n",
17981939
" self.word_embeds = nn.Embedding(vocab_size, embedding_dim)\n",
17991940
" self.lstm = nn.LSTM(embedding_dim, hidden_dim/2, num_layers=1, bidirectional=True)\n",
1941+
" \n",
1942+
" # Maps the output of the LSTM into tag space.\n",
18001943
" self.hidden2tag = nn.Linear(hidden_dim, self.tagset_size)\n",
1944+
" \n",
1945+
" # Matrix of transition parameters. Entry i,j is the score of transitioning *to* i *from* j.\n",
18011946
" self.transitions = nn.Parameter(torch.randn(self.tagset_size, self.tagset_size))\n",
18021947
" \n",
18031948
" self.hidden = self.init_hidden()\n",
@@ -1808,17 +1953,26 @@
18081953
" \n",
18091954
" \n",
18101955
" def _forward_alg(self, feats):\n",
1956+
" # Do the forward algorithm to compute the partition function\n",
18111957
" init_alphas = torch.Tensor(1, self.tagset_size).fill_(-10000.)\n",
1958+
" # START_TAG has all of the score.\n",
18121959
" init_alphas[0][self.tag_to_ix[START_TAG]] = 0.\n",
18131960
" \n",
1961+
" # Wrap in a variable so that we will get automatic backprop\n",
18141962
" forward_var = autograd.Variable(init_alphas)\n",
18151963
" \n",
1964+
" # Iterate through the sentence\n",
18161965
" for feat in feats:\n",
1817-
" alphas_t = []\n",
1966+
" alphas_t = [] # The forward variables at this timestep\n",
18181967
" for next_tag in xrange(self.tagset_size):\n",
1968+
" # broadcast the emission score: it is the same regardless of the previous tag\n",
18191969
" emit_score = feat[next_tag].expand(self.tagset_size)\n",
1970+
" # the ith entry of trans_score is the score of transitioning to next_tag from i\n",
18201971
" trans_score = self.transitions[next_tag]\n",
1972+
" # The ith entry of next_tag_var is the value for the edge (i -> next_tag)\n",
1973+
" # before we do log-sum-exp\n",
18211974
" next_tag_var = forward_var + trans_score + emit_score\n",
1975+
" # The forward variable for this tag is log-sum-exp of all the scores.\n",
18221976
" alphas_t.append(log_sum_exp(next_tag_var))\n",
18231977
" forward_var = torch.cat(alphas_t).view(1, -1)\n",
18241978
" terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]\n",
@@ -1833,6 +1987,7 @@
18331987
" return lstm_feats\n",
18341988
" \n",
18351989
" def _score_sentence(self, feats, tags):\n",
1990+
" # Gives the score of a provided tag sequence\n",
18361991
" score = autograd.Variable( torch.Tensor([0]) )\n",
18371992
" tags = [self.tag_to_ix[START_TAG]] + tags\n",
18381993
" for i, feat in enumerate(feats):\n",
@@ -1870,18 +2025,21 @@
18702025
" best_path.reverse()\n",
18712026
" return best_path, path_score\n",
18722027
" \n",
1873-
" def log_likelihood(self, sentence, tags):\n",
2028+
" def neg_log_likelihood(self, sentence, tags):\n",
18742029
" feats = self._get_lstm_features(sentence)\n",
18752030
" forward_score = self._forward_alg(feats)\n",
18762031
" gold_score = self._score_sentence(feats, tags)\n",
1877-
" return gold_score - forward_score\n",
2032+
" return -(gold_score - forward_score)\n",
18782033
" \n",
1879-
" def forward(self, sentence):\n",
2034+
" def forward(self, sentence): # dont confuse this with _forward_alg above.\n",
18802035
" embeds = self.word_embeds(sentence).view(len(sentence), 1, -1)\n",
2036+
" # Get the emission features from the LSTM\n",
18812037
" lstm_out, self.hidden = self.lstm(embeds)\n",
18822038
" lstm_out = lstm_out.view(len(sentence), self.hidden_dim)\n",
2039+
" # Map into tag space\n",
18832040
" lstm_feats = self.hidden2tag(lstm_out)\n",
18842041
" \n",
2042+
" # Find the best path, given the features.\n",
18852043
" tag_seq, score = self._viterbi_decode(lstm_feats)\n",
18862044
" return score, tag_seq\n",
18872045
"\n",
@@ -1894,21 +2052,12 @@
18942052
"model = BiLSTM_CRF(2, tag_to_ix, 4, 6)\n",
18952053
"model(idxs)\n"
18962054
]
1897-
},
1898-
{
1899-
"cell_type": "code",
1900-
"execution_count": null,
1901-
"metadata": {
1902-
"collapsed": true
1903-
},
1904-
"outputs": [],
1905-
"source": []
19062055
}
19072056
],
19082057
"metadata": {
19092058
"kernelspec": {
19102059
"display_name": "Python 2",
1911-
"language": "python2",
2060+
"language": "python",
19122061
"name": "python2"
19132062
},
19142063
"language_info": {

‎README.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,10 @@ There are plenty of other tutorials out there, but they all seem to have one of
1818
* Exercise: Continuous Bag-of-Words for learning word embeddings
1919
7. Sequence modeling and Long-Short Term Memory Networks
2020
* Example: An LSTM for Part-of-Speech Tagging
21-
* Exercise: LSTM Language Modeling
21+
* Exercise: Augmenting the LSTM tagger with character-level features
2222
8. Advanced: Making Dynamic Decisions
2323
* Example: Bi-LSTM Conditional Random Field for named-entity recognition
24-
24+
2525
# To do:
2626
* Add decoding to the LSTM POS tagger example
2727
* Comment the Bi LSTM CRF example and provide more discussion
28-
* Write the LSTM LM exercise (I might change this: there are tons of LSTM LM examples out there...)

0 commit comments

Comments
 (0)