harry's blog

1. Dropout

torch.nn.Dropout(p=0.5, inplace=False)

p – probability of an element to be zeroed. Default: 0.5
inplace – If set to True, will do this operation in-place. Default: False

训练过程中以概率 P 随机的将参数置 0，其中 P 为置 0 的概率，例如 P=1 表示将网络参数全部置 0

During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.

注意: Pytorch 文档中给出了一点，输出的参数会以$\frac{1}{1-p}$进行一个缩放

Furthermore, the outputs are scaled by a factor of $\frac{1}{1-p}$ during training. This means that during evaluation the module simply computes an identity function.

下面例子展示出在 dropout 之后，参数变为了原来的$\frac{1}{1-p} = 2$倍

input = torch.tensor([[1, 2, 3],
                      [4, 5, 6],
                      [7, 8, 9]], dtype=torch.float64)
input = torch.unsqueeze(input, 0)
m = nn.Dropout(p = 0.5)
output = m(input)

print("input: ", input)
print("output: ", output)
print("input: ", input)
'''
input:  
tensor([[[1., 2., 3.],
         [4., 5., 6.],
         [7., 8., 9.]]], dtype=torch.float64)
output:  
tensor([[[ 2.,  4.,  0.],
         [ 0., 10., 12.],
         [ 0., 16.,  0.]]], dtype=torch.float64)
input:  
tensor([[[1., 2., 3.],
         [4., 5., 6.],
         [7., 8., 9.]]], dtype=torch.float64)
'''

当我们把 nn.Dropout 的 inplace=True 时，计算的结果就会替换掉原来的输入 input，如下：

input = torch.tensor([[1, 2, 3],
                      [4, 5, 6],
                      [7, 8, 9]], dtype=torch.float64)
input = torch.unsqueeze(input, 0)
m = nn.Dropout(p = 0.5, inplace=True)
output = m(input)

print("input: ", input)
print("output: ", output)
print("input: ", input)
'''
input:  
tensor([[[1., 2., 3.],
         [4., 5., 6.],
         [7., 8., 9.]]], dtype=torch.float64)
output:  
tensor([[[ 2.,  4.,  0.],
         [ 0., 10., 12.],
         [ 0., 16.,  0.]]], dtype=torch.float64)
input:  
tensor([[[ 2.,  4.,  0.],
         [ 0., 10., 12.],
         [ 0., 16.,  0.]]], dtype=torch.float64)
'''

1.1 训练与测试的不同

在训练和测试的时候，nn.Dropout 的表现是不同的，在训练时 nn.Dropout 会以概率 p 随机的丢弃一些神经元，但是在测试时，所有神经元都不会被丢弃，如下

import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self, p=0.0):
        super().__init__()
        self.drop_layer = nn.Dropout(p=p)

    def forward(self, inputs):
        return self.drop_layer(inputs)
        
model = Model(p=0.5) # functional dropout
# creating inputs
inputs = torch.rand(10)
# forwarding inputs in train mode
print('Normal (train) model:')
print('Model ', model(inputs))

# switching to eval mode
model.eval()
# forwarding inputs in evaluation mode
print('Evaluation mode:')
print('Model ', model(inputs))
# show model summary
print('Print summary:')
print(model)
'''
Normal (train) model:
Model  tensor([0.0000, 1.3517, 0.0000, 0.2766, 0.3060, 1.6334, 0.0000, 0.9740, 0.9118,
        0.0000])

Evaluation mode:
Model  tensor([0.9284, 0.6758, 0.3947, 0.1383, 0.1530, 0.8167, 0.2038, 0.4870, 0.4559,
        0.2730])
Print summary:
Model(
  (drop_layer): Dropout(p=0.5, inplace=False)
)
'''

本文由 Yonghui Wang 创作，采用知识共享署名4.0 国际许可协议进行许可
本站文章除注明转载/出处外，均为本站原创或翻译，转载前请务必署名
最后编辑时间为: Dec 19, 2024 12:18 pm