What I have in mind is changing:
DateUtils.getRelativeTimeSpanString(due, now, 0L, DateUtils.FORMAT_ABBREV_ALL);
Since the documentation says it returns the time relative to now.
If that fails use some of the brilliant libraries:
Joda Time
PrettyTime
TimeAgo