tokenizer_encoder_plus_token_type_ids

加载模型

from transformers import BertTokenizer

model_names = 'bert-base-uncased'

tokenizer = BertTokenizer.from_pretrained(model_name)

tokenizer

tokenizer.special_tokens_map

special_tokens = list(tokenizer.special_tokens_map.values())

specical_tokens

tokenizer.convert_tokens_to_ids(special_tokens)

tokenizer.encode(special_tokens)

tokenizer.decode(...)

认识文本语料

获得语料信息

• newsgroups_train.DESCR
• newsgroups_train.data
• newsgroups_train.target
• newsgroups_train.target_names

from sklearn.datasets import fetch_20newsgroups

newsgroups_train = fetch_20newsgroups(subset='train')

newsgroups_train.DESCR

from collections import Counter

Counter(newsgroups_train.target)

newsgroups_train.target_names

tokenizer补充

• input_ids,attention_masks
• encode_plus,token_type_ids
• token_type_ids —> 表示第一句:1;表示第2句:0
• 句子对，一般用在nsp(next sentence predict)级别的，bert训练的1个预训练任务

test_news = newsgroups_train.data[:3]

len(test_news[2])

truncation截断

#single sentence级别
tokenizer(test_news,truncation=True,max_lengths=32)

#sentence pair级别的
tokenizer.encode_plus(text=text_news[0],text_pair=text_news[1])

tokenizer.decode