9 views (last 30 days)

Show older comments

Joseph Conroy about 8 hours ago

Commented: Matt J about 6 hours ago

Open in MATLAB Online

Key questions:

- Why does a network predict a specific value for the output regardless of input as if the input data had no information relevant to prediction?
- Why does replacing min-max scaling with standard scaling fix this, at least occassionally?

The problem background: I am trying to train a simple image autoencoder, but I keep getting networks that only output a single image regardless of the input. Taking the difference between each output image reveals they are all exactly the same. Googling this issue, I saw a stack overflow post that this often arises with improperly dimensioned loss functions. I also saw folks mentioning issues with using the sigmoid loss function for autoencoders, but the explanations as to why never surpass guesswork. I changed the scaling from min-max scaling to standard scaling and was able to obtain a network that breaks out of the single-prediction behavior, but without understanding why, I will have no recourse but trial-and-error if it breaks again.

Notes on dimensioning loss functions: When calculating the loss between a batch of images of shape [imgDim, imgDim, 1, batchSize] the mse loss function outputs a loss of dimension [1,1,1,batchSize], but this loss function has produced defective results under min-max scaling, such as the aforementioned degeneration to a single output, as well as an initial loss three orders of magnitude above the inputs and outputs scaled to the range [0,1]. To be clear, I don't mean the learning is unstable, I mean that the absolute values of the loss are absurd.

I tried to write my own loss function that reports a scalar value, but I encountered the same degeneration to a single prediction independent of input. I then wrote a version that reports an error tensor of the same shape as @mse, but this threw an error listed below, after the custom loss function in question.

% Version that reports a scalar

function meanAbsErr = myMae(prediction, target)

meanAbsErr = mean(abs(flatten(prediction) - flatten(target)), 'all');

end

% Version that reports [1,1,1,batchSize]

function meanAbsErr = myMae(prediction, target)

inDims = size(prediction);

meanAbsErr = mean(abs(flatten(prediction) - flatten(target)), 1);

outDims = ones(1,length(inDims)); outDims(end) = inDims(end);

meanAbsErr = reshape(meanAbsErr, outDims);

end

Value to differentiate is non-scalar. It must be a traced real dlarray scalar.

Error in mathworksDebug>modelLoss (line 213)

[gradientsE,gradientsD] = dlgradient(loss,netE.Learnables,netD.Learnables);

Error in deep.internal.dlfeval (line 17)

[varargout{1:nargout}] = fun(x{:});

Error in deep.internal.dlfevalWithNestingCheck (line 19)

[varargout{1:nargout}] = deep.internal.dlfeval(fun,varargin{:});

Error in dlfeval (line 31)

[varargout{1:nargout}] = deep.internal.dlfevalWithNestingCheck(fun,varargin{:});

Error in mathworksDebug (line 134)

[loss,gradientsE,gradientsD] = dlfeval(@modelLoss,netE,netD,X,Ztarget);

Notes on scaling

I wrote a custom scaling function that executes the same behavior as rescale except that it reports the obtained extrema to use in scaling and de-scaling unseen data.

% Min-max scaling between [lb, ub]

function [scaled,smin,smax] = myRescale(varargin)

datastruct = varargin{1}; lb = varargin{2}; ub = varargin{3};

if length(varargin) <= 3

smin = min(datastruct(:)); smax = max(datastruct(:));

else

smin = varargin{4}; smax = varargin{5};

end

scaled = (datastruct - smin) / (smax - smin) * (ub - lb) + lb;

end

% Invert scaling

function unscaled = myDescale(scaled, lb, ub, smin, smax)

unscaled = (scaled + lb ) * (smax - smin) ./ (ub - lb) + smin;

end

% Converts the data to z-scores

function [standard, center, stddev] = myStandardize(varargin)

datastruct = varargin{1};

if length(varargin) == 1

center = mean(datastruct(:)); stddev = std(datastruct(:));

else

center = varargin{2}; stddev = varargin{3};

end

standard = (datastruct - center) / stddev;

end

% Converts z-scores back to the data's scale

function destandard = myDestandardize(datastruct, center, stddev)

destandard = datastruct * stddev + center;

end

In the following code, I have removed the validation set to reduce bloat.

% % I intend to regularize the latent space of this autoencoder to be a

% classify images once it can accomplish basic reconstruction. I made this note so

% it's clear what's going on with the custom losses and so forth.

xTrain = digitTrain4DArrayData;

xTest = digitTest4DArrayData;

%% Scaling that does not work

% Min-max scaling

xlb = 0; xub=1;

[xTrain, xTrainMin, xTrainMax] = myRescale(xTrain, xl, xub);

xTest = myRescale(xTest, xTrainMin, xTrainMax);

%% Scaling that works, at least occasionally

xTest = myStandardize(xTest, xTrainCenter, xTrainStd);

ntrain = size(xDev,4);

IMG_DIM = size(xDev, 1);N_CHANNELS=size(xDev, 3);

OUT_CHANNELS = min(size(tTrain,1), 64);

numLatentChannels = OUT_CHANNELS;

imageSize = [28 28 1];

%% Layer definitions

% Encoder layer

layersE = [

imageInputLayer(imageSize,Normalization="none")

convolution2dLayer(3,32,Padding="same",Stride=2)

reluLayer

convolution2dLayer(3,64,Padding="same",Stride=2)

reluLayer

fullyConnectedLayer(numLatentChannels)

tanhLayer(Name='latent')];

% Latent projection

projectionSize = [7 7 64]; enc_dim = projectionSize(1);

numInputChannels = imageSize(3);

% Decoder

layersD = [

featureInputLayer(numLatentChannels)

projectAndReshapeLayer(projectionSize)

transposedConv2dLayer(3,64,Cropping="same",Stride=2)

reluLayer

transposedConv2dLayer(3,32,Cropping="same",Stride=2)

reluLayer

transposedConv2dLayer(3,numInputChannels,Cropping="same")

sigmoidLayer('Output')

];

netE = dlnetwork(layersE);

netD = dlnetwork(layersD);

%% Training Parameters

numEpochs = 150;

miniBatchSize = 20;

learnRate = 1e-3;

dsXTrain = arrayDatastore(xTrain,IterationDimension=4);

dstTrain = arrayDatastore(tTrain,IterationDimension=2);

numOutputs = 2;

dsTrain = combine(dsXTrain, dstTrain);

mbq = minibatchqueue(dsTrain,numOutputs, ...

MiniBatchSize = miniBatchSize, ...

MiniBatchFormat=["SSCB", "CB"], ...

MiniBatchFcn=@preprocessMiniBatch,...

PartialMiniBatch="return");

%Initialize the parameters for the Adam solver.

trailingAvgE = [];

trailingAvgSqE = [];

trailingAvgD = [];

trailingAvgSqD = [];

%Calculate the total number of iterations for the training progress monitor

numIterationsPerEpoch = ceil(ntrain / miniBatchSize);

numIterations = numEpochs * numIterationsPerEpoch;

epoch = 0;

%Initialize the training progress monitor.

monitor = trainingProgressMonitor( ...

Metrics=["TrainingLoss"], ...

Info=["Epoch", "LearningRate"], ...

XLabel="Iteration");

%% Training

while epoch < numEpochs && ~monitor.Stop

epoch = epoch + 1;

% Shuffle data.

shuffle(mbq);

% Loop over mini-batches.

while hasdata(mbq) && ~monitor.Stop

iteration = iteration + 1;

% Read mini-batch of data.

[X, Ztarget] = next(mbq);

% Evaluate loss and gradients.

[loss,gradientsE,gradientsD] = dlfeval(@modelLoss,netE,netD,X,Ztarget);

% Update learnable parameters.

[netE,trailingAvgE,trailingAvgSqE] = adamupdate(netE, ...

gradientsE,trailingAvgE,trailingAvgSqE,iteration,learnRate);

[netD, trailingAvgD, trailingAvgSqD] = adamupdate(netD, ...

gradientsD,trailingAvgD,trailingAvgSqD,iteration,learnRate);

updateInfo(monitor, ...

LearningRate=learnRate, ...

Epoch=string(epoch) + " of " + string(numEpochs));

recordMetrics(monitor,iteration, ...

TrainingLoss=loss);

monitor.Progress = 100*iteration/numIterations;

end

end

%% Testing

dsTest = combine(arrayDatastore(xTest,IterationDimension=4),...

arrayDatastore(tTest,IterationDimension=2));

numOutputs = 2;

mbqTest = minibatchqueue(dsTest,numOutputs, ...

MiniBatchSize = miniBatchSize, ...

MiniBatchFcn=@preprocessMiniBatch, ...

MiniBatchFormat="SSCB");

[YTest, ZTest] = modelPredictions(netE,netD,mbqTest);

reconerr = mean(flatten(xTest-YTest),1);

figure

histogram(reconerr)

xlabel("Reconstruction Error")

ylabel("Frequency")

title("Test Data")

numImages = 64;

ndisplay = 10;

figure

I = imtile(YTest(:,:,:,1:numImages));

imshow(I)

title("Reconstructed Images")

%% Functions

function [loss,gradientsE,gradientsD] = modelLoss(netE,netD,X,Ztarget)

% Forward through encoder.

Z = forward(netE,X);

% Forward through decoder.

Xrecon = forward(netD,Z);

% Calculate loss and gradients.

loss = regularizedLoss(Xrecon,X,Z,Ztarget);

[gradientsE,gradientsD] = dlgradient(loss,netE.Learnables,netD.Learnables);

end

function loss = regularizedLoss(Xrecon,X,Z,Ztarget)

% Image Reconstruction loss.

reconstructionLoss = mse(Xrecon, X);

% Regularized Loss

%regLoss = mse(Z, Ztarget);

% Combined loss.

loss = reconstructionLoss;% + 0.0*regLoss;

end

function [Xrecon, Zpred] = modelPredictions(netE,netD,mbq)

Xrecon = [];

Zpred = [];

% Loop over mini-batches.

while hasdata(mbq)

X = next(mbq);

% Pass through encoder

Z = predict(netE,X);

% Pass through decoder to get reconstructed images

XGenerated = predict(netD,Z);

% Extract and concatenate predictions.

Xrecon = cat(4,Xrecon,extractdata(XGenerated));

Zpred = cat(2,Zpred,extractdata(Z));

end

end

function loss = assessLoss(netE, netD, X, Ztarget)

% Forward through encoder.

Z = predict(netE,X);

% Forward through decoder.

Xrecon = predict(netD,Z);

% Calculate loss and gradients.

loss = regularizedLoss(Xrecon,X,Z,Ztarget);

end

function [X, Ztarget] = preprocessMiniBatch(Xcell, tCell)

% Concatenate.

X = cat(4,Xcell{:});

% Concatenate.

Ztarget = cat(2,tCell{:});

end

##### 1 Comment Show -1 older commentsHide -1 older comments

Show -1 older commentsHide -1 older comments

Matt J 6 minutes ago

#### Direct link to this comment

https://au.mathworks.com/matlabcentral/answers/2136633-why-is-this-autoencoder-only-predicting-a-single-output-regardless-of-input-when-using-min-max-scali#comment_3209258

Your code doesn't show what the training targets are, and hwo they are processed. For all we know, you could be using the same target image over and over again by mistake.

Sign in to comment.

Sign in to answer this question.

### Answers (0)

Sign in to answer this question.

**An Error Occurred**

Unable to complete the action because of changes made to the page. Reload the page to see its updated state.

Select a Web Site

Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .

You can also select a web site from the following list

Americas

- América Latina (Español)
- Canada (English)
- United States (English)

Europe

- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)

- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- Deutsch
- English
- Français

- United Kingdom(English)

Asia Pacific

- Australia (English)
- India (English)
- New Zealand (English)
- 中国
- 日本Japanese (日本語)
- 한국Korean (한국어)

Contact your local office