You can use the sliding function from MLLIB which probably does the same thing as Daniel’s answer. You will have to sort the data by time before using the sliding function. import org.apache.spark.mllib.rdd.RDDFunctions._ sc.parallelize(1 to 100, 10) .sliding(3) .map(curSlice => (curSlice.sum / curSlice.size)) .collect()
Matplotlib uses an internal floating point format for dates. You just need to convert your date to that format (using matplotlib.dates.date2num or matplotlib.dates.datestr2num) and then use annotate as usual. As a somewhat excessively fancy example: import datetime as dt import matplotlib.pyplot as plt import matplotlib.dates as mdates x = [dt.datetime(2009, 05, 01), dt.datetime(2010, 06, 01), … Read more
You can use dt.floor for convert to dates and then value_counts or groupby with size: df = (pd.to_datetime(df[‘date & time of connection’]) .dt.floor(‘d’) .value_counts() .rename_axis(‘date’) .reset_index(name=”count”)) print (df) date count 0 2017-06-23 6 1 2017-06-21 5 2 2017-06-19 3 3 2017-06-22 3 4 2017-06-20 2 Or: s = pd.to_datetime(df[‘date & time of connection’]) df = … Read more
You can use pd.DateOffset: test[1].index + pd.DateOffset(hours=16) pd.DateOffset accepts the same keyword arguments as dateutil.relativedelta. The problem you encountered was due to this bug which has been fixed in Pandas version 0.14.1: In [242]: pd.to_timedelta(16, unit=”h”) Out[242]: numpy.timedelta64(16,’ns’) If you upgrade, your original code should work.
Looks like you want to use the Series.shift method. Using this method, you can generate new columns which are offset to the original columns. Like this: df[‘qty_s’] = df[‘qty’].shift(-1) df[‘t_s’] = df[‘t’].shift(-1) df[‘z_s’] = df[‘z’].shift(-1) Now you can compare these: df[‘is_something’] = (df[‘qty’] == df[‘qty_s’]) & (df[‘t’] < df[‘t_s’]) & (df[‘z’] == df[‘z_s’]) Here is … Read more
Matplotlib uses an internal floating point format for dates. You just need to convert your date to that format (using matplotlib.dates.date2num or matplotlib.dates.datestr2num) and then use annotate as usual. As a somewhat excessively fancy example: import datetime as dt import matplotlib.pyplot as plt import matplotlib.dates as mdates x = [dt.datetime(2009, 05, 01), dt.datetime(2010, 06, 01), … Read more
With pandas 0.18 the resample API changed (see the docs). So for pandas >= 0.18 the answer is: In [31]: frame.resample(‘1H’).agg({‘radiation’: np.sum, ‘tamb’: np.mean}) Out[31]: tamb radiation 2012-04-05 08:00:00 5.161235 279.507182 2012-04-05 09:00:00 4.968145 290.941073 2012-04-05 10:00:00 4.478531 317.678285 2012-04-05 11:00:00 4.706206 335.258633 2012-04-05 12:00:00 2.457873 8.655838 Old Answer: I am answering my question to … Read more
According to Wikipedia, EWMA works well with stationary data, but it does not work as expected in the presence of trends, or seasonality. In those cases you should use a second or third order EWMA method, respectively. I decided to look at the pandas ewma function to see how it handled trends, and this is … Read more