dataframe

Spark add new column to dataframe with value from previous row

by

You can use lag window function as follows from pyspark.sql.functions import lag, col from pyspark.sql.window import Window df = sc.parallelize([(4, 9.0), (3, 7.0), (2, 3.0), (1, 5.0)]).toDF([“id”, “num”]) w = Window().partitionBy().orderBy(col(“id”)) df.select(“*”, lag(“num”).over(w).alias(“new_col”)).na.drop().show() ## +—+—+——-+ ## | id|num|new_col| ## +—+—+——-| ## | 2|3.0| 5.0| ## | 3|7.0| 3.0| ## | 4|9.0| 7.0| ## +—+—+——-+ but … Read more

Dropping time from datetime

by

The quickest way is to use DatetimeIndex’s normalize (you first need to make the column a DatetimeIndex): In [11]: df = pd.DataFrame({“t”: pd.date_range(‘2014-01-01′, periods=5, freq=’H’)}) In [12]: df Out[12]: t 0 2014-01-01 00:00:00 1 2014-01-01 01:00:00 2 2014-01-01 02:00:00 3 2014-01-01 03:00:00 4 2014-01-01 04:00:00 In [13]: pd.DatetimeIndex(df.t).normalize() Out[13]: <class ‘pandas.tseries.index.DatetimeIndex’> [2014-01-01, …, 2014-01-01] Length: … Read more

404 Not Found

Not Found

The requested URL was not found on this server.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.