這篇文章給大家分享的是有關(guān)Pytorch如何實現(xiàn)數(shù)據(jù)加載與數(shù)據(jù)預(yù)處理的內(nèi)容。小編覺得挺實用的，因此分享給大家做個參考，一起跟隨小編過來看看吧。
數(shù)據(jù)加載分為加載torchvision.datasets中的數(shù)據(jù)集以及加載自己使用的數(shù)據(jù)集兩種情況。
torchvision.datasets中的數(shù)據(jù)集
torchvision.datasets中自帶MNIST，Imagenet-12，CIFAR等數(shù)據(jù)集，所有的數(shù)據(jù)集都是torch.utils.data.Dataset的子類，都包含 _ _ len _ （獲取數(shù)據(jù)集長度）和 _ getItem _ _ （獲取數(shù)據(jù)集中每一項）兩個子方法。
Dataset源碼如上，可以看到其中包含了兩個沒有實現(xiàn)的子方法，之后所有的Dataet類都繼承該類，并根據(jù)數(shù)據(jù)情況定制這兩個子方法的具體實現(xiàn)。
因此當我們需要加載自己的數(shù)據(jù)集的時候也可以借鑒這種方法，只需要繼承torch.utils.data.Dataset類并重寫 init ,len,以及getitem這三個方法即可。這樣組著的類可以直接作為參數(shù)傳入到torch.util.data.DataLoader中去。
以CIFAR10為例 源碼：
class torchvision.datasets.CIFAR10(root, train=True, transform=None, target_transform=None, download=False)
root (string) – Root directory of dataset where directory cifar-10-batches-py exists or will be saved to if download is set to True. train (bool, optional) – If True, creates dataset from training set, otherwise creates from test set. transform (callable, optional) – A function/transform that takes in an PIL image and returns a transformed version. E.g, transforms.RandomCrop target_transform (callable, optional) – A function/transform that takes in the target and transforms it. download (bool, optional) – If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again.
加載自己的數(shù)據(jù)集
對于torchvision.datasets中有兩個不同的類，分別為DatasetFolder和ImageFolder,ImageFolder是繼承自DatasetFolder。
下面我們通過源碼來看一看folder文件中DatasetFolder和ImageFolder分別做了些什么
重慶分公司
重慶分公司
import torch.utils.data as data
from PIL import Image
import os
import os.path
def has_file_allowed_extension(filename, extensions): //檢查輸入是否是規(guī)定的擴展名
"""Checks if a file is an allowed extension.
Args:
filename (string): path to a file
Returns:
bool: True if the filename ends with a known image extension
"""
filename_lower = filename.lower()
return any(filename_lower.endswith(ext) for ext in extensions)
def find_classes(dir):
classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))] //獲取root目錄下所有的文件夾名稱
classes.sort()
class_to_idx = {classes[i]: i for i in range(len(classes))} //生成類別名稱與類別id的對應(yīng)Dictionary
return classes, class_to_idx
def make_dataset(dir, class_to_idx, extensions):
images = []
dir = os.path.expanduser(dir)// 將~和~user轉(zhuǎn)化為用戶目錄，對參數(shù)中出現(xiàn)~進行處理
for target in sorted(os.listdir(dir)):
d = os.path.join(dir, target)
if not os.path.isdir(d):
continue
for root, _, fnames in sorted(os.walk(d)): //os.work包含三個部分，root代表該目錄路徑 _代表該路徑下的文件夾名稱集合，fnames代表該路徑下的文件名稱集合
for fname in sorted(fnames):
if has_file_allowed_extension(fname, extensions):
path = os.path.join(root, fname)
item = (path, class_to_idx[target])
images.append(item) //生成（訓(xùn)練樣本圖像目錄，訓(xùn)練樣本所屬類別）的元組
return images //返回上述元組的列表
class DatasetFolder(data.Dataset):
"""A generic data loader where the samples are arranged in this way: ::
root/class_x/xxx.ext
root/class_x/xxy.ext
root/class_x/xxz.ext
root/class_y/123.ext
root/class_y/nsdf3.ext
root/class_y/asd932_.ext
Args:
root (string): Root directory path.
loader (callable): A function to load a sample given its path.
extensions (list[string]): A list of allowed extensions.
transform (callable, optional): A function/transform that takes in
a sample and returns a transformed version.
E.g, ``transforms.RandomCrop`` for images.
target_transform (callable, optional): A function/transform that takes
in the target and transforms it.
Attributes:
classes (list): List of the class names.
class_to_idx (dict): Dict with items (class_name, class_index).
samples (list): List of (sample path, class_index) tuples
"""
def __init__(self, root, loader, extensions, transform=None, target_transform=None):
classes, class_to_idx = find_classes(root)
samples = make_dataset(root, class_to_idx, extensions)
if len(samples) == 0:
raise(RuntimeError("Found 0 files in subfolders of: " + root + "\n"
"Supported extensions are: " + ",".join(extensions)))
self.root = root
self.loader = loader
self.extensions = extensions
self.classes = classes
self.class_to_idx = class_to_idx
self.samples = samples
self.transform = transform
self.target_transform = target_transform
def __getitem__(self, index):
"""
根據(jù)index獲取sample 返回值為（sample，target）元組，同時如果該類輸入?yún)?shù)中有transform和target_transform，torchvision.transforms類型的參數(shù)時，將獲取的元組分別執(zhí)行transform和target_transform中的數(shù)據(jù)轉(zhuǎn)換方法。
Args:
index (int): Index
Returns:
tuple: (sample, target) where target is class_index of the target class.
"""
path, target = self.samples[index]
sample = self.loader(path)
if self.transform is not None:
sample = self.transform(sample)
if self.target_transform is not None:
target = self.target_transform(target)
return sample, target
def __len__(self):
return len(self.samples)
def __repr__(self): //定義輸出對象格式 其中和__str__的區(qū)別是__repr__無論是print輸出還是直接輸出對象自身 都是以定義的格式進行輸出，而__str__ 只有在print輸出的時候會是以定義的格式進行輸出
fmt_str = 'Dataset ' + self.__class__.__name__ + '\n'
fmt_str += ' Number of datapoints: {}\n'.format(self.__len__())
fmt_str += ' Root Location: {}\n'.format(self.root)
tmp = ' Transforms (if any): '
fmt_str += '{0}{1}\n'.format(tmp, self.transform.__repr__().replace('\n', '\n' + ' ' * len(tmp)))
tmp = ' Target Transforms (if any): '
fmt_str += '{0}{1}'.format(tmp, self.target_transform.__repr__().replace('\n', '\n' + ' ' * len(tmp)))
return fmt_str
IMG_EXTENSIONS = ['.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif']
def pil_loader(path):
# open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
with open(path, 'rb') as f:
img = Image.open(f)
return img.convert('RGB')
def accimage_loader(path):
import accimage
try:
return accimage.Image(path)
except IOError:
# Potentially a decoding problem, fall back to PIL.Image
return pil_loader(path)
def default_loader(path):
from torchvision import get_image_backend
if get_image_backend() == 'accimage':
return accimage_loader(path)
else:
return pil_loader(path)
class ImageFolder(DatasetFolder):
"""A generic data loader where the images are arranged in this way: ::
root/dog/xxx.png
root/dog/xxy.png
root/dog/xxz.png
root/cat/123.png
root/cat/nsdf3.png
root/cat/asd932_.png
Args:
root (string): Root directory path.
transform (callable, optional): A function/transform that takes in an PIL image
and returns a transformed version. E.g, ``transforms.RandomCrop``
target_transform (callable, optional): A function/transform that takes in the
target and transforms it.
loader (callable, optional): A function to load an image given its path.
Attributes:
classes (list): List of the class names.
class_to_idx (dict): Dict with items (class_name, class_index).
imgs (list): List of (image path, class_index) tuples
"""
def __init__(self, root, transform=None, target_transform=None,
loader=default_loader):
super(ImageFolder, self).__init__(root, loader, IMG_EXTENSIONS,
transform=transform,
target_transform=target_transform)
self.imgs = self.samples如果自己所要加載的數(shù)據(jù)組織形式如下
root/dog/xxx.png root/dog/xxy.png root/dog/xxz.png root/cat/123.png root/cat/nsdf3.png root/cat/asd932_.png
即不同類別的訓(xùn)練數(shù)據(jù)分別存儲在不同的文件夾中，這些文件夾都在root（即形如 D:/animals 或者 /usr/animals ）路徑下
