tsvcsvtxtjson格式文件的处理方法是怎样的
tsv csv txt json格式文件的处理方法是怎样的,相信很多没有经验的人对此束手无策,为此本文总结了问题出现的原因和解决方法,通过这篇文章希望你能解决这个问题。
创新互联公司于2013年创立,先为甘肃等服务建站,甘肃等地企业,进行企业商务咨询服务。为甘肃企业网站制作PC+手机+微官网三网同步一站式服务解决您的所有建站问题。
对于tsv、csv、txt以及json类型的数据的处理方法一般可以使用torchtext中的TabularDataset进行处理;
数据的要求:
tsv: 第一行fields字段名,使用tab隔开,其它行为数据,每个字段直接的数据使用tab隔开;
csv: 第一行fields字段,其它行为数据
json: 字典类型,每一行为一个字典,字典的key为fields,values为数据。
本次采用以下tsv格式的数据集:
sentiment-analysis-on-movie-reviews.zip
数据集的格式:
注意:如果test数据集中缺少某些字段,使用torchtext处理时会有问题,因此要保证train val和test数据集要处理的字段必需相同。
方法一: torchtext
任务:构造一个翻译类型的数据集
inputs:[sequence english] target:[sequence chinese]
from torchtext.data import Field, TabularDataset, BucketIterator import torch batch_size = 6 device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') tokenize_x = lambda x: x.split() tokenize_y = lambda y: y TEXT = Field(sequential=True, use_vocab=True, tokenize=tokenize_x, lower=True, batch_first=True, init_token='', eos_token=' ') LABEL = Field(sequential=False, use_vocab=False, tokenize=tokenize_y, batch_first=True, init_token=None, eos_token=None) # fields = {'english': ('en', ENGLISH), 'chinese': ('cn', CHINESE)} # The first of element tuple was tsv's fields_name fields = [("PhraseId", None), ("SentenceId", None), ("Phrase", TEXT), ("Sentiment", LABEL)] train_data, test_data = TabularDataset.splits(path='data', train='movie-sentiment_train.tsv', test='movie-sentiment_test.tsv', format='tsv', skip_header=True, fields=fields) TEXT.build_vocab(train_data, max_size=10000, min_freq=2) VOCAB_SIZE = len(TEXT.vocab) # The operation of vocabulary print("vocabulary size: ", VOCAB_SIZE) print(TEXT.vocab.freqs) print(TEXT.vocab.itos[:10]) for i, v in enumerate(TEXT.vocab.stoi): if i == 10: break print(v) print(TEXT.vocab.stoi['apple']) print(' indx is ', TEXT.vocab.stoi[' ']) print(' indx is ', TEXT.vocab.stoi[' ']) UNK_STR = TEXT.unk_token PAD_STR = TEXT.pad_token UNK_IDX = TEXT.vocab.stoi[UNK_STR] PAD_IDX = TEXT.vocab.stoi[PAD_STR] print(f'{UNK_STR} index is {UNK_IDX}') print(f'{PAD_STR} index is {PAD_IDX}') # The operation of datasets print(len(train_data)) print(train_data[0].__dict__.keys()) print(train_data[0].__dict__.values()) # vars return attribute of object print(vars(train_data.examples[0])) print(train_data[0].Phrase) print(train_data[0].Sentiment) """ batch_sizes: Tuple of batch sizes to use for the different splits, or None to use the same batch_size for all splits. """ train_iterator, test_iterator = BucketIterator.splits((train_data, test_data), batch_size=32, batch_sizes=None, device=device, repeat=False, # shuffle=True, sort_key=lambda x: len(x.Phrase), sort=False, sort_within_batch=True) for batch in train_iterator: print(batch.Phrase.shape) print([TEXT.vocab.itos[idx] for idx in batch.Phrase[0]]) print(batch.Sentiment) break
如果只有一个文本数据需要处理,将splits方法去除,修改以下初始化参数,修改的代码如下:
fields = [("PhraseId", None), ("SentenceId", None), ("Phrase", TEXT), ("Sentiment", LABEL)] train_data = TabularDataset(path='data/movie-sentiment_train.tsv', format='tsv', skip_header=True, fields=fields) train_iterator = BucketIterator(train_data, batch_size=batch_size, device=device, shuffle=False, repeat = False, sort_key=lambda x: len(x.Phrase), sort_within_batch=False)
fields是否需要use_vocab为True,即是否需要建立一个字典:
对于inputs数据而言,都需要进行词典的建立,而对于labels而言,如果labels是数字类型的数据(实际是string类型),通常在iterator会使用int()强制转换成longTensor()类型,如果labels不是数字类型的数据,需要建立一个词典,这样在iterator会字段转换成longTensor类型。
关于TabularDataset中fieds字段传入list和dict的区别:
list
构造fields时必须按照数据集中fields字段的顺序依次构造,优点: 数据集第一行可以不写字段名,缺点:train test val数据集所有字段必须完全相同。
TabularDataset中skip_header字段要根据数据集的第一行是否有fields名称设置成True或者False。
fields = [("PhraseId", None), ("SentenceId", None), ("Phrase", TEXT), ("Sentiment", LABEL)]
dict
构造fields时可以根据自己的需要选择性的选择字段,优点:train test val数据集所有字段可以不完全相同,缺点:数据集的第一行必须有字段名称。
TabularDataset中skip_header字段必须是False。
fields = {'Phrase': ('Phrase', TEXT), 'Sentiment': ('Sentiment', LABEL)}
BucketIterator中sort和shuffle问题:
shuffle参数用于是否打乱每个batch的取出顺序,推荐使用默认参数,即train数据集打乱,其它数据集不打乱;
sort_key=lambda x: len(x.Phrase): 按照何种方式排序;
sort: 对所有数据集进行降序排序;推荐False.
sort_within_batch:对每个batch进行升序排序;推荐使用True.
方法二:手撕代码
任务:构造一个翻译类型的数据集
inputs:[english, chinese] target:[(english, en_len, chinese, cn_len), (...)]
步骤:
分词生成两维的列表
分别创建词典
根据词典使用索引替换英文和中文词
构造batch
根据英文句子个数和batchSize构造batch的索引组
根据创建的batch索引,构造batch数据,并返回每句话的长度list
import torch import numpy as np import nltk import jieba from collections import Counter UNK_IDX = 0 PAD_IDX = 1 batch_size = 64 train_file = 'data/translate_train.txt' dev_file = 'data/translate_dev.txt' """ 数据格式: english \t chinese 读取英文中文翻译文件, 句子开头和结尾分别加上返回两个列表 """ def load_data(in_file): cn = [] en = [] with open(in_file, 'r', encoding='utf-8') as f: for line in f: line = line.strip().split("\t") en.append(['BOS'] + nltk.word_tokenize(line[0].lower()) + ['EOS']) # cn.append(['BOS'] + [c for c in line[1]] + ['EOS']) cn.append(['BOS'] + jieba.lcut(line[1]) + ['EOS']) return en, cn """ 创建词典 """ def build_dict(sentences, max_words=50000): word_count = Counter() for sentence in sentences: for s in sentence: word_count[s] += 1 ls = word_count.most_common(max_words) total_words = len(ls) + 2 word_dict = {w[0]: index for index, w in enumerate(ls, 2)} word_dict['UNK'] = UNK_IDX word_dict['PAD'] = PAD_IDX return word_dict, total_words # 把句子变成索引 def encode(en_sentences, cn_sentences, en_dict, cn_dict, sort_by_len=True): """ Encode the sequences. """ length = len(en_sentences) # 将句子的词转换成词典对应的索引 out_en_sentences = [[en_dict.get(w, 0) for w in sent] for sent in en_sentences] out_cn_sentences = [[cn_dict.get(w, 0) for w in sent] for sent in cn_sentences] def len_argsort(seq): return sorted(range(len(seq)), key=lambda x: len(seq[x])) if sort_by_len: sorted_index = len_argsort(out_en_sentences) out_en_sentences = [out_en_sentences[i] for i in sorted_index] out_cn_sentences = [out_cn_sentences[i] for i in sorted_index] return out_en_sentences, out_cn_sentences def get_minibatches(n, minibatch_size, shuffle=False): idx_list = np.arange(0, n, minibatch_size) # [0, 1, ..., n-1] if shuffle: np.random.shuffle(idx_list) minibatches = [] for idx in idx_list: minibatches.append(np.arange(idx, min(idx + minibatch_size, n))) return minibatches def prepare_data(seqs, padding_idx): lengths = [len(seq) for seq in seqs] n_samples = len(seqs) max_len = np.max(lengths) x = np.full((n_samples, max_len), padding_idx).astype('int32') x_lengths = np.array(lengths).astype("int32") for idx, seq in enumerate(seqs): x[idx, :lengths[idx]] = seq return x, x_lengths #x_mask def gen_examples(en_sentences, cn_sentences, batch_size): minibatches = get_minibatches(len(en_sentences), batch_size) all_ex = [] for minibatch in minibatches: mb_en_sentences = [en_sentences[t] for t in minibatch] mb_cn_sentences = [cn_sentences[t] for t in minibatch] mb_x, mb_x_len = prepare_data(mb_en_sentences, PAD_IDX) mb_y, mb_y_len = prepare_data(mb_cn_sentences, PAD_IDX) all_ex.append((mb_x, mb_x_len, mb_y, mb_y_len)) return all_ex train_en, train_cn = load_data(train_file) dev_en, dev_cn = load_data(dev_file) en_dict, en_total_words = build_dict(train_en) cn_dict, cn_total_words = build_dict(train_cn) inv_en_dict = {v: k for k, v in en_dict.items()} inv_cn_dict = {v: k for k, v in cn_dict.items()} train_en, train_cn = encode(train_en, train_cn, en_dict, cn_dict) dev_en, dev_cn = encode(dev_en, dev_cn, en_dict, cn_dict) print(" ".join([inv_cn_dict[i] for i in train_cn[100]])) print(" ".join([inv_en_dict[i] for i in train_en[100]])) train_data = gen_examples(train_en, train_cn, batch_size) dev_data = gen_examples(dev_en, dev_cn, batch_size) print(len(train_data)) print(train_data[0])
看完上述内容,你们掌握tsv csv txt json格式文件的处理方法是怎样的的方法了吗?如果还想学到更多技能或想了解更多相关内容,欢迎关注创新互联行业资讯频道,感谢各位的阅读!
网站题目:tsvcsvtxtjson格式文件的处理方法是怎样的
文章网址:http://scyanting.com/article/iegeeg.html