用于摘要、情感分类和翻译的 T5-Base 模型

作者: Pendo Abbo，Joe Cummings

概述

本教程演示了如何使用预训练的 T5 模型进行摘要、情感分类和翻译任务。我们将演示如何使用 torchtext 库来

1. 为 T5 模型构建文本预处理管道

2. 使用基础配置实例化预训练的 T5 模型

3. 读取 CNNDM、IMDB 和 Multi30k 数据集，并预处理其文本以备模型使用

4. 执行文本摘要、情感分类和翻译

数据转换

T5 模型不适用于原始文本。相反，它需要将文本转换为数值形式才能执行训练和推理。T5 模型需要以下转换

1. 分词

2. 将词元转换为（整数）ID

3. 将序列截断到指定的最大长度

4. 添加序列结束 (EOS) 和填充词元 ID

T5 使用 SentencePiece 模型进行文本分词。下面，我们使用预训练的 SentencePiece 模型使用 torchtext 的 T5Transform 构建文本预处理管道。请注意，转换支持批处理和非批处理文本输入（例如，可以传递单个句子或句子列表），但是 T5 模型期望输入为批处理。

from torchtext.models import T5Transform

padding_idx = 0
eos_idx = 1
max_seq_len = 512
t5_sp_model_path = "https://download.pytorch.org/models/text/t5_tokenizer_base.model"

transform = T5Transform(
    sp_model_path=t5_sp_model_path,
    max_seq_len=max_seq_len,
    eos_idx=eos_idx,
    padding_idx=padding_idx,
)

或者，我们也可以使用预训练模型附带的转换，它可以开箱即用地完成所有上述操作

from torchtext.models import T5_BASE_GENERATION
transform = T5_BASE_GENERATION.transform()

模型准备

torchtext 提供了 SOTA 预训练模型，可以直接用于 NLP 任务或微调到下游任务。下面，我们使用具有标准基础配置的预训练 T5 模型来执行文本摘要、情感分类和翻译。有关可用预训练模型的更多详细信息，请参阅 torchtext 文档

from torchtext.models import T5_BASE_GENERATION


t5_base = T5_BASE_GENERATION
transform = t5_base.transform()
model = t5_base.get_model()
model.eval()

GenerationUtils

我们可以使用 torchtext 的 GenerationUtils 根据提供的输入序列生成输出序列。这调用模型的编码器和解码器，并迭代地扩展解码序列，直到为批次中的所有序列生成序列结束词元。下面显示的 generate 方法使用贪婪搜索生成序列。还支持集束搜索和其他解码策略。

from torchtext.prototype.generate import GenerationUtils

sequence_generator = GenerationUtils(model)

数据集

torchtext 提供了几个标准的 NLP 数据集。有关完整列表，请参阅 https://pytorch.ac.cn/text/stable/datasets.html 上的文档。这些数据集使用可组合的 torchdata 数据管道构建，因此支持使用用户定义函数和转换进行标准流程控制和映射/转换。

下面，我们演示如何预处理 CNNDM 数据集以包含模型识别其正在执行的任务所需的 prefix。CNNDM 数据集具有训练、验证和测试拆分。下面，我们在测试拆分上进行演示。

T5 模型使用 prefix “summarize” 进行文本摘要。有关任务 prefix 的更多信息，请访问 T5 论文 的附录 D

注意

使用数据管道目前仍存在一些注意事项。如果您希望将此示例扩展到包括洗牌、多处理或分布式学习，请参阅 此注释 以获取更多说明。

from functools import partial

from torch.utils.data import DataLoader
from torchtext.datasets import CNNDM

cnndm_batch_size = 5
cnndm_datapipe = CNNDM(split="test")
task = "summarize"


def apply_prefix(task, x):
    return f"{task}: " + x[0], x[1]


cnndm_datapipe = cnndm_datapipe.map(partial(apply_prefix, task))
cnndm_datapipe = cnndm_datapipe.batch(cnndm_batch_size)
cnndm_datapipe = cnndm_datapipe.rows2columnar(["article", "abstract"])
cnndm_dataloader = DataLoader(cnndm_datapipe, shuffle=True, batch_size=None)

或者，我们也可以使用批处理 API，例如，在整个批次上应用 prefix

def batch_prefix(task, x):
 return {
     "article": [f'{task}: ' + y for y in x["article"]],
     "abstract": x["abstract"]
 }

cnndm_batch_size = 5
cnndm_datapipe = CNNDM(split="test")
task = 'summarize'

cnndm_datapipe = cnndm_datapipe.batch(cnndm_batch_size).rows2columnar(["article", "abstract"])
cnndm_datapipe = cnndm_datapipe.map(partial(batch_prefix, task))
cnndm_dataloader = DataLoader(cnndm_datapipe, batch_size=None)

我们还可以加载 IMDB 数据集，该数据集将用于演示使用 T5 模型进行情感分类。此数据集具有训练和测试拆分。下面，我们在测试拆分上进行演示。

T5 模型在 SST2 数据集（也存在于 torchtext 中）上进行了训练，以使用 prefix “sst2 sentence” 进行情感分类。因此，我们将使用此 prefix 对 IMDB 数据集执行情感分类。

from torchtext.datasets import IMDB

imdb_batch_size = 3
imdb_datapipe = IMDB(split="test")
task = "sst2 sentence"
labels = {"1": "negative", "2": "positive"}


def process_labels(labels, x):
    return x[1], labels[str(x[0])]


imdb_datapipe = imdb_datapipe.map(partial(process_labels, labels))
imdb_datapipe = imdb_datapipe.map(partial(apply_prefix, task))
imdb_datapipe = imdb_datapipe.batch(imdb_batch_size)
imdb_datapipe = imdb_datapipe.rows2columnar(["text", "label"])
imdb_dataloader = DataLoader(imdb_datapipe, batch_size=None)

最后，我们还可以加载 Multi30k 数据集以演示使用 T5 模型进行英语到德语的翻译。此数据集具有训练、验证和测试拆分。下面，我们在测试拆分上进行演示。

T5 模型为此任务使用 prefix “translate English to German”。

from torchtext.datasets import Multi30k

multi_batch_size = 5
language_pair = ("en", "de")
multi_datapipe = Multi30k(split="test", language_pair=language_pair)
task = "translate English to German"

multi_datapipe = multi_datapipe.map(partial(apply_prefix, task))
multi_datapipe = multi_datapipe.batch(multi_batch_size)
multi_datapipe = multi_datapipe.rows2columnar(["english", "german"])
multi_dataloader = DataLoader(multi_datapipe, batch_size=None)

生成摘要

我们可以将所有组件放在一起，使用光束大小为 1 在 CNNDM 测试集中第一批文章上生成摘要。

batch = next(iter(cnndm_dataloader))
input_text = batch["article"]
target = batch["abstract"]
beam_size = 1

model_input = transform(input_text)
model_output = sequence_generator.generate(model_input, eos_idx=eos_idx, num_beams=beam_size)
output_text = transform.decode(model_output.tolist())

for i in range(cnndm_batch_size):
    print(f"Example {i+1}:\n")
    print(f"prediction: {output_text[i]}\n")
    print(f"target: {target[i]}\n\n")

摘要输出（由于我们对数据加载器进行了洗牌，因此可能会有所不同）

Example 1:

prediction: the 24-year-old has been tattooed for over a decade . he has landed in australia
to start work on a new campaign . he says he is 'taking it in your stride' to be honest .

target: London-based model Stephen James Hendry famed for his full body tattoo . The supermodel
is in Sydney for a new modelling campaign . Australian fans understood to have already located
him at his hotel . The 24-year-old heartthrob is recently single .


Example 2:

prediction: a stray pooch has used up at least three of her own after being hit by a
car and buried in a field . the dog managed to stagger to a nearby farm, dirt-covered
and emaciated, where she was found . she suffered a dislocated jaw, leg injuries and a
caved-in sinus cavity -- and still requires surgery to help her breathe .

target: Theia, a bully breed mix, was apparently hit by a car, whacked with a hammer
and buried in a field . "She's a true miracle dog and she deserves a good life," says
Sara Mellado, who is looking for a home for Theia .


Example 3:

prediction: mohammad Javad Zarif arrived in Iran on a sunny friday morning . he has gone
a long way to bring Iran in from the cold and allow it to rejoin the international
community . but there are some facts about him that are less well-known .

target: Mohammad Javad Zarif has spent more time with John Kerry than any other
foreign minister . He once participated in a takeover of the Iranian Consulate in San
Francisco . The Iranian foreign minister tweets in English .


Example 4:

prediction: five americans were monitored for three weeks after being exposed to Ebola in
west africa . one of the five had a heart-related issue and has been discharged but hasn't
left the area . they are clinicians for Partners in Health, a Boston-based aid group .

target: 17 Americans were exposed to the Ebola virus while in Sierra Leone in March .
Another person was diagnosed with the disease and taken to hospital in Maryland .
National Institutes of Health says the patient is in fair condition after weeks of
treatment .


Example 5:

prediction: the student was identified during an investigation by campus police and
the office of student affairs . he admitted to placing the noose on the tree early
Wednesday morning . the incident is one of several recent racist events to affect
college students .

target: Student is no longer on Duke University campus and will face disciplinary
review . School officials identified student during investigation and the person
admitted to hanging the noose, Duke says . The noose, made of rope, was discovered on
campus about 2 a.m.

生成情感分类

类似地，我们可以使用模型在 IMDB 测试集中第一批评论上生成情感分类，光束大小为 1。

batch = next(iter(imdb_dataloader))
input_text = batch["text"]
target = batch["label"]
beam_size = 1

model_input = transform(input_text)
model_output = sequence_generator.generate(model_input, eos_idx=eos_idx, num_beams=beam_size)
output_text = transform.decode(model_output.tolist())

for i in range(imdb_batch_size):
    print(f"Example {i+1}:\n")
    print(f"input_text: {input_text[i]}\n")
    print(f"prediction: {output_text[i]}\n")
    print(f"target: {target[i]}\n\n")

情感输出

Example 1:

input_text: sst2 sentence: I love sci-fi and am willing to put up with a lot. Sci-fi
movies/TV are usually underfunded, under-appreciated and misunderstood. I tried to like
this, I really did, but it is to good TV sci-fi as Babylon 5 is to Star Trek (the original).
Silly prosthetics, cheap cardboard sets, stilted dialogues, CG that doesn't match the
background, and painfully one-dimensional characters cannot be overcome with a 'sci-fi'
setting. (I'm sure there are those of you out there who think Babylon 5 is good sci-fi TV.
It's not. It's clichéd and uninspiring.) While US viewers might like emotion and character
development, sci-fi is a genre that does not take itself seriously (cf. Star Trek). It may
treat important issues, yet not as a serious philosophy. It's really difficult to care about
the characters here as they are not simply foolish, just missing a spark of life. Their
actions and reactions are wooden and predictable, often painful to watch. The makers of Earth
KNOW it's rubbish as they have to always say "Gene Roddenberry's Earth..." otherwise people
would not continue watching. Roddenberry's ashes must be turning in their orbit as this dull,
cheap, poorly edited (watching it without advert breaks really brings this home) trudging
Trabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bring
him back as another actor. Jeeez. Dallas all over again.

prediction: negative

target: negative


Example 2:

input_text: sst2 sentence: Worth the entertainment value of a rental, especially if you like
action movies. This one features the usual car chases, fights with the great Van Damme kick
style, shooting battles with the 40 shell load shotgun, and even terrorist style bombs. All
of this is entertaining and competently handled but there is nothing that really blows you
away if you've seen your share before.<br /><br />The plot is made interesting by the
inclusion of a rabbit, which is clever but hardly profound. Many of the characters are
heavily stereotyped -- the angry veterans, the terrified illegal aliens, the crooked cops,
the indifferent feds, the bitchy tough lady station head, the crooked politician, the fat
federale who looks like he was typecast as the Mexican in a Hollywood movie from the 1940s.
All passably acted but again nothing special.<br /><br />I thought the main villains were
pretty well done and fairly well acted. By the end of the movie you certainly knew who the
good guys were and weren't. There was an emotional lift as the really bad ones got their just
deserts. Very simplistic, but then you weren't expecting Hamlet, right? The only thing I found
really annoying was the constant cuts to VDs daughter during the last fight scene.<br /><br />
Not bad. Not good. Passable 4.

prediction: positive

target: negative


Example 3:

input_text: sst2 sentence: its a totally average film with a few semi-alright action sequences
that make the plot seem a little better and remind the viewer of the classic van dam films.
parts of the plot don't make sense and seem to be added in to use up time. the end plot is that
of a very basic type that doesn't leave the viewer guessing and any twists are obvious from the
beginning. the end scene with the flask backs don't make sense as they are added in and seem to
have little relevance to the history of van dam's character. not really worth watching again,
bit disappointed in the end production, even though it is apparent it was shot on a low budget
certain shots and sections in the film are of poor directed quality.

prediction: negative

target: negative

生成翻译

最后，我们还可以使用模型在 Multi30k 测试集中第一批示例上生成英语到德语的翻译。

batch = next(iter(multi_dataloader))
input_text = batch["english"]
target = batch["german"]

model_input = transform(input_text)
model_output = sequence_generator.generate(model_input, eos_idx=eos_idx, num_beams=beam_size)
output_text = transform.decode(model_output.tolist())

for i in range(multi_batch_size):
    print(f"Example {i+1}:\n")
    print(f"input_text: {input_text[i]}\n")
    print(f"prediction: {output_text[i]}\n")
    print(f"target: {target[i]}\n\n")

翻译输出

Example 1:

input_text: translate English to German: A man in an orange hat starring at something.

prediction: Ein Mann in einem orangen Hut, der an etwas schaut.

target: Ein Mann mit einem orangefarbenen Hut, der etwas anstarrt.


Example 2:

input_text: translate English to German: A Boston Terrier is running on lush green grass in front of a white fence.

prediction: Ein Boston Terrier läuft auf üppigem grünem Gras vor einem weißen Zaun.

target: Ein Boston Terrier läuft über saftig-grünes Gras vor einem weißen Zaun.


Example 3:

input_text: translate English to German: A girl in karate uniform breaking a stick with a front kick.

prediction: Ein Mädchen in Karate-Uniform bricht einen Stöck mit einem Frontkick.

target: Ein Mädchen in einem Karateanzug bricht ein Brett mit einem Tritt.


Example 4:

input_text: translate English to German: Five people wearing winter jackets and helmets stand in the snow, with snowmobiles in the background.

prediction: Fünf Menschen mit Winterjacken und Helmen stehen im Schnee, mit Schneemobilen im Hintergrund.

target: Fünf Leute in Winterjacken und mit Helmen stehen im Schnee mit Schneemobilen im Hintergrund.


Example 5:

input_text: translate English to German: People are fixing the roof of a house.

prediction: Die Leute fixieren das Dach eines Hauses.

target: Leute Reparieren das Dach eines Hauses.