jQuery append fadeIn

Your first attempt is very close, but remember that append() is returning #thumbnails, not the item you just added to it. Instead, construct your item first and apply the hide().fadeIn() before adding it:

$('#thumbnails')
    .append($('<li><img src="/photos/t/'+data.filename+'"/></li>')
        .hide()
        .fadeIn(2000)
    );

This uses the dollar function to construct the <li> ahead of time. You could also write it on two lines, of course, if that makes it clearer:

var item = $('<li><img src="/photos/t/'+data.filename+'"/></li>')
    .hide()
    .fadeIn(2000);
$('#thumbnails').append(item);

Edit: Your second attempt is also almost there, but you need to use children() instead of filter(). The latter only removes nodes from the current query; your newly-added item isn’t in that query, but is a child node instead.

$('#thumbnails')
    .append('<li style="display:none"><img src="/photos/t/'+data.filename+'"/></li>')
    .children(':last')
    .hide()
    .fadeIn(2000);