Pyspark UDF

PHOTO

Mon Dec 12 2022 15:41:23 GMT+0000 (Coordinated Universal Time)

import pyspark.sql.types as T
@udf(returnType=T.ArrayType(T.ArrayType(T.DoubleType())))
def aad():
	return [[1.22]]

df.withColumn("new_col", aad(F.col()))

COPY

https://sparkbyexamples.com/pyspark/pyspark-udf-user-defined-function/

Save snippets that work from anywhere online with our extensions

Comments

More like this

#pyspark #spark #python #etl

Split Spark Dataframe string column into multiple columns

split_col = pyspark.sql.functions.split(df['my_str_col'], '-')
df = df.withColumn('NAME1', split_col.getItem(0))
df = df.withColumn('NAME2', split_col.getItem(1))

#python #pyspark #spark #spark-session

Create a basic pyspark session

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('abc').getOrCreate()

#pyspark #unique #count #dataframe #column

pyspark dataframe column unique count

trx_1.select(f.countDistinct("stg_nexus_member_cd")).show()

#pyspark #python #panda

Date Range - PySpark and Pandas API

>>> ps.date_range(start='1/1/2018', periods=5, freq='M')  
DatetimeIndex(['2018-01-31', '2018-02-28', '2018-03-31', '2018-04-30',
               '2018-05-31'],
              dtype='datetime64[ns]', freq=None)

#pyspark #python

PySpark GroupBy multiple columns


df.groupBy("Add","Name").agg({'id':'sum'}).show()

#pyspark #python

Get specific row from PySpark dataframe

# get first row
print(dataframe.collect()[0])
  
# get second row
print(dataframe.collect()[1])
  
# get last row
print(dataframe.collect()[-1])
  
# get third row
print(dataframe.collect()[2])

#udf #pyspark

Pyspark UDF

import pyspark.sql.types as T
@udf(returnType=T.ArrayType(T.ArrayType(T.DoubleType())))
def aad():
	return [[1.22]]

df.withColumn("new_col", aad(F.col()))

Pyspark UDF

Save snippets that work from anywhere online with our extensions

Comments

More like this

Browse more snippets >>

Embed code snippet