Calculate window function over timestamp of seconds

PHOTO EMBED

Mon Aug 15 2022 19:56:05 GMT+0000 (Coordinated Universal Time)

Saved by @Milados

hours = lambda i: i * 60*60
w = (Window.partitionBy([col(x) for x in cols + []]).orderBy(col('Raised').cast("timestamp").cast("long")).rangeBetween(-hours(4), 0))
df = df.withColumn('SumAlarms', count("Rack").over(w))
content_copyCOPY

+-----------+----+-----+--------+--------+--------------+---------+-------------------+ |CLLI |Rack|Shelf|OLT_Slot|OLT_Port|ONT_Identifier|SumAlarms|Raised | +-----------+----+-----+--------+--------+--------------+---------+-------------------+ |ADLNCAXFOL2|R1 |S1 |LT10 |PON3 |ONT22 |1 |2021-11-16 03:07:29| |ADLNCAXFOL2|R1 |S1 |LT10 |PON4 |ONT7 |1 |2022-01-29 16:11:50| |ADLNCAXFOL2|R1 |S1 |LT10 |PON4 |ONT7 |2 |2022-01-29 17:05:01| |ADLNCAXFOL2|R1 |S1 |LT10 |PON4 |ONT7 |1 |2022-03-09 05:31:28| |ADLNCAXFOL2|R1 |S1 |LT10 |PON4 |ONT7 |2 |2022-03-09 05:31:43| |ADLNCAXFOL2|R1 |S1 |LT10 |PON4 |ONT7 |1 |2022-04-13 12:13:17| |ADLNCAXFOL2|R1 |S1 |LT10 |PON4 |ONT7 |1 |2022-04-14 12:48:17| |ADLNCAXFOL2|R1 |S1 |LT10 |PON4 |ONT7 |2 |2022-04-14 16:45:53| |ADLNCAXFOL2|R1 |S1 |LT10 |PON4 |ONT7 |1 |2022-04-18 12:27:29| |ADLNCAXFOL2|R1 |S1 |LT10 |PON4 |ONT7 |2 |2022-04-18 15:17:02| |ADLNCAXFOL2|R1 |S1 |LT10 |PON4 |ONT7 |1 |2022-04-20 11:02:42| |ADLNCAXFOL2|R1 |S1 |LT10 |PON4 |ONT7 |1 |2022-05-01 16:35:46|

http://localhost:8888/notebooks/01-EDA-network.ipynb