I got what i wanted eventually using this workaround. I hope it helps whoever might need something similar.
Stack(
children: <Widget>[
Positioned(
left: 1.0,
top: 2.0,
child: Icon(icon, color: Colors.black54),
),
Icon(icon, color: Colors.white),
],
),