注意
点击此处下载完整示例代码
序列模型和长短期记忆网络¶
创建日期:2017 年 4 月 8 日 | 最后更新:2022 年 1 月 7 日 | 最后验证:未验证
至此,我们已经了解了各种前馈网络。也就是说,网络根本不维护任何状态。这可能不是我们想要的行为。序列模型是 NLP 的核心:它们是输入之间存在某种时间依赖关系的模型。序列模型的典型例子是用于词性标注的隐马尔可夫模型。另一个例子是条件随机场。
循环神经网络是一种维护某种状态的网络。例如,其输出可以作为下一个输入的一部分,以便信息在网络处理序列时沿途传播。对于 LSTM,序列中的每个元素都有一个对应的隐藏状态 \(h_t\),原则上它可以包含序列中任意早前点的信息。我们可以使用隐藏状态来预测语言模型中的词语、词性标签以及许多其他事物。
PyTorch 中的 LSTM¶
在进入示例之前,请注意一些事项。PyTorch 的 LSTM 要求其所有输入都是 3D 张量。这些张量轴的语义很重要。第一轴是序列本身,第二轴索引 mini-batch 中的实例,第三轴索引输入元素。我们尚未讨论 mini-batch,所以暂且忽略它,并假设第二轴始终只有 1 个维度。如果我们要对句子“The cow jumped”运行序列模型,我们的输入应该看起来像
但请记住,还有一个额外的第 2 维度,大小为 1。
此外,您可以一次处理序列中的一个元素,在这种情况下,第 1 轴的大小也将为 1。
让我们看一个快速示例。
# Author: Robert Guthrie
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
torch.manual_seed(1)
<torch._C.Generator object at 0x7f5713d9a470>
lstm = nn.LSTM(3, 3) # Input dim is 3, output dim is 3
inputs = [torch.randn(1, 3) for _ in range(5)] # make a sequence of length 5
# initialize the hidden state.
hidden = (torch.randn(1, 1, 3),
torch.randn(1, 1, 3))
for i in inputs:
# Step through the sequence one element at a time.
# after each step, hidden contains the hidden state.
out, hidden = lstm(i.view(1, 1, -1), hidden)
# alternatively, we can do the entire sequence all at once.
# the first value returned by LSTM is all of the hidden states throughout
# the sequence. the second is just the most recent hidden state
# (compare the last slice of "out" with "hidden" below, they are the same)
# The reason for this is that:
# "out" will give you access to all hidden states in the sequence
# "hidden" will allow you to continue the sequence and backpropagate,
# by passing it as an argument to the lstm at a later time
# Add the extra 2nd dimension
inputs = torch.cat(inputs).view(len(inputs), 1, -1)
hidden = (torch.randn(1, 1, 3), torch.randn(1, 1, 3)) # clean out hidden state
out, hidden = lstm(inputs, hidden)
print(out)
print(hidden)
tensor([[[-0.0187, 0.1713, -0.2944]],
[[-0.3521, 0.1026, -0.2971]],
[[-0.3191, 0.0781, -0.1957]],
[[-0.1634, 0.0941, -0.1637]],
[[-0.3368, 0.0959, -0.0538]]], grad_fn=<MkldnnRnnLayerBackward0>)
(tensor([[[-0.3368, 0.0959, -0.0538]]], grad_fn=<StackBackward0>), tensor([[[-0.9825, 0.4715, -0.0633]]], grad_fn=<StackBackward0>))
示例:用于词性标注的 LSTM¶
在本节中,我们将使用 LSTM 获取词性标签。我们不会使用 Viterbi 或 Forward-Backward 等算法,但作为一项(有挑战性的)练习,在您了解正在发生的事情后,可以思考如何使用 Viterbi。在此示例中,我们还提及了词嵌入(embeddings)。如果您不熟悉词嵌入,可以在此处阅读相关内容。
模型如下:假设输入句子为 \(w_1, \dots, w_M\),其中 \(w_i \in V\) 是我们的词汇表。另外,设 \(T\) 是我们的标签集,\(y_i\) 是词语 \(w_i\) 的标签。将我们对词语 \(w_i\) 标签的预测表示为 \(\hat{y}_i\)。
这是一个结构化预测模型,其输出是序列 \(\hat{y}_1, \dots, \hat{y}_M\),其中 \(\hat{y}_i \in T\)。
为了进行预测,对句子应用一个 LSTM。将时间步 \(i\) 的隐藏状态表示为 \(h_i\)。此外,为每个标签分配一个唯一索引(就像我们在词嵌入章节中的 word_to_ix 一样)。那么,我们对 \(\hat{y}_i\) 的预测规则是
也就是说,取隐藏状态经过仿射映射后的 log softmax,预测的标签就是该向量中具有最大值的标签。请注意,这立即表明 \(A\) 的目标空间的维度是 \(|T|\)。
准备数据
def prepare_sequence(seq, to_ix):
idxs = [to_ix[w] for w in seq]
return torch.tensor(idxs, dtype=torch.long)
training_data = [
# Tags are: DET - determiner; NN - noun; V - verb
# For example, the word "The" is a determiner
("The dog ate the apple".split(), ["DET", "NN", "V", "DET", "NN"]),
("Everybody read that book".split(), ["NN", "V", "DET", "NN"])
]
word_to_ix = {}
# For each words-list (sentence) and tags-list in each tuple of training_data
for sent, tags in training_data:
for word in sent:
if word not in word_to_ix: # word has not been assigned an index yet
word_to_ix[word] = len(word_to_ix) # Assign each word with a unique index
print(word_to_ix)
tag_to_ix = {"DET": 0, "NN": 1, "V": 2} # Assign each tag with a unique index
# These will usually be more like 32 or 64 dimensional.
# We will keep them small, so we can see how the weights change as we train.
EMBEDDING_DIM = 6
HIDDEN_DIM = 6
{'The': 0, 'dog': 1, 'ate': 2, 'the': 3, 'apple': 4, 'Everybody': 5, 'read': 6, 'that': 7, 'book': 8}
创建模型
class LSTMTagger(nn.Module):
def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
super(LSTMTagger, self).__init__()
self.hidden_dim = hidden_dim
self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)
# The LSTM takes word embeddings as inputs, and outputs hidden states
# with dimensionality hidden_dim.
self.lstm = nn.LSTM(embedding_dim, hidden_dim)
# The linear layer that maps from hidden state space to tag space
self.hidden2tag = nn.Linear(hidden_dim, tagset_size)
def forward(self, sentence):
embeds = self.word_embeddings(sentence)
lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))
tag_space = self.hidden2tag(lstm_out.view(len(sentence), -1))
tag_scores = F.log_softmax(tag_space, dim=1)
return tag_scores
训练模型
model = LSTMTagger(EMBEDDING_DIM, HIDDEN_DIM, len(word_to_ix), len(tag_to_ix))
loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)
# See what the scores are before training
# Note that element i,j of the output is the score for tag j for word i.
# Here we don't need to train, so the code is wrapped in torch.no_grad()
with torch.no_grad():
inputs = prepare_sequence(training_data[0][0], word_to_ix)
tag_scores = model(inputs)
print(tag_scores)
for epoch in range(300): # again, normally you would NOT do 300 epochs, it is toy data
for sentence, tags in training_data:
# Step 1. Remember that Pytorch accumulates gradients.
# We need to clear them out before each instance
model.zero_grad()
# Step 2. Get our inputs ready for the network, that is, turn them into
# Tensors of word indices.
sentence_in = prepare_sequence(sentence, word_to_ix)
targets = prepare_sequence(tags, tag_to_ix)
# Step 3. Run our forward pass.
tag_scores = model(sentence_in)
# Step 4. Compute the loss, gradients, and update the parameters by
# calling optimizer.step()
loss = loss_function(tag_scores, targets)
loss.backward()
optimizer.step()
# See what the scores are after training
with torch.no_grad():
inputs = prepare_sequence(training_data[0][0], word_to_ix)
tag_scores = model(inputs)
# The sentence is "the dog ate the apple". i,j corresponds to score for tag j
# for word i. The predicted tag is the maximum scoring tag.
# Here, we can see the predicted sequence below is 0 1 2 0 1
# since 0 is index of the maximum value of row 1,
# 1 is the index of maximum value of row 2, etc.
# Which is DET NOUN VERB DET NOUN, the correct sequence!
print(tag_scores)
tensor([[-1.1389, -1.2024, -0.9693],
[-1.1065, -1.2200, -0.9834],
[-1.1286, -1.2093, -0.9726],
[-1.1190, -1.1960, -0.9916],
[-1.0137, -1.2642, -1.0366]])
tensor([[-0.0462, -4.0106, -3.6096],
[-4.8205, -0.0286, -3.9045],
[-3.7876, -4.1355, -0.0394],
[-0.0185, -4.7874, -4.6013],
[-5.7881, -0.0186, -4.1778]])
练习:使用字符级特征增强 LSTM 词性标注器¶
在上面的示例中,每个词语都有一个词嵌入,作为我们序列模型的输入。让我们使用从词语字符派生的表示来增强词嵌入。我们期望这将显著有所帮助,因为像词缀这样的字符级信息对词性有很大的影响。例如,带有词缀 -ly 的词语在英语中几乎总是被标记为副词。
为此,设 \(c_w\) 是词语 \(w\) 的字符级表示。设 \(x_w\) 是之前的词嵌入。那么我们序列模型的输入就是 \(x_w\) 和 \(c_w\) 的拼接。因此,如果 \(x_w\) 的维度是 5,\(c_w\) 的维度是 3,那么我们的 LSTM 应该接受一个维度为 8 的输入。
要获得字符级表示,对词语的字符应用一个 LSTM,并让 \(c_w\) 成为此 LSTM 的最终隐藏状态。提示
您的新模型中将有两个 LSTM。原始的那个输出词性标签分数,新的那个输出每个词语的字符级表示。
要对字符构建序列模型,您需要嵌入字符。字符嵌入将是字符 LSTM 的输入。
脚本总运行时间: ( 0 分钟 0.506 秒)