Given this df: date 0 2001-08-10 1 2002-08-31 2 2003-08-29 3 2006-06-21 4 2002-03-27 5 2003-07-14 6 2004-06-15 7 2003-08-14 8 2003-07-29 and, if it’s not already the case: df[“date”] = df[“date”].astype(“datetime64”) To show the count of dates by month: df.groupby(df[“date”].dt.month).count().plot(kind=”bar”) .dt allows you to access the datetime properties. Which will give you: You can … Read more
Can be done without conversion to/from int (but to/from timestamp instead) SELECT date_trunc(‘day’, dd):: date FROM generate_series ( ‘2007-02-01’::timestamp , ‘2008-04-01’::timestamp , ‘1 day’::interval) dd ;
It’s worth mentioning that you may have been able to read this in directly e.g. if you were using read_csv using parse_dates=[[‘Date’, ‘Time’]]. Assuming these are just strings you could simply add them together (with a space), allowing you to use to_datetime, which works without specifying the format= parameter In [11]: df[‘Date’] + ‘ ‘ … Read more
Definitely Relational. Unlimited flexibility and expansion. Two corrections, both in concept and application, followed by an elevation. Correction It is not “filtering out the un-needed data”; it is selecting only the needed data. Yes, of course, if you have an Index to support the columns identified in the WHERE clause, it is very fast, and … Read more
If you just want a straightforward non-weighted moving average, you can easily implement it with np.cumsum, which may be is faster than FFT based methods: EDIT Corrected an off-by-one wrong indexing spotted by Bean in the code. EDIT def moving_average(a, n=3) : ret = np.cumsum(a, dtype=float) ret[n:] = ret[n:] – ret[:-n] return ret[n – 1:] … Read more
For a small number of variables, you can build the plot manually yourself: ggplot(test_data, aes(date)) + geom_line(aes(y = var0, colour = “var0”)) + geom_line(aes(y = var1, colour = “var1”))
Robust peak detection algorithm (using z-scores) I came up with an algorithm that works very well for these types of datasets. It is based on the principle of dispersion: if a new datapoint is a given x number of standard deviations away from some moving mean, the algorithm signals (also called z-score). The algorithm is … Read more