Flatten Json string using explode

 Using PySpark to Read and Flatten JSON data with an enforced schema – Ben Alex Keen


data_df1 = spark.read.json(df.rdd.map( lambda row:row.body), schema=sensor_schema)

from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType, ArrayType

sensor_schema = StructType(fields=[
    StructField('accelerometer_x', StringType(), True),
    StructField('accelerometer_y', StringType(), True),
    StructField('accelerometer_z',StringType(),True),
    StructField('gyroscope',
    StructType(fields=[
        StructField('gyroscope_x',StringType(),True),
        StructField('gyroscope_y',StringType(),True),
        StructField('gyroscope_z',StringType(),True)
    ]))
        
])

data_df2 = spark.read.json(df.rdd.map( lambda row:row.body), schema=sensor_schema)
display(data_df2)
data_df2.printSchema()

data_df1=data_df2.select(col('accelerometer_x'),col('accelerometer_y'),col('accelerometer_z'),col('gyroscope.gyroscope_x'),col('gyroscope.gyroscope_y'),col('gyroscope.gyroscope_z'))
display(data_df1)


Comments

Post a Comment

Popular posts from this blog

Introduction To Oracle10g

                                                  Oracle Database 10 g, released in 2003 and the current release, enables grid (the g in 10g) computing. A grid is simply a pool of computers that provides needed resources for applications on an as-needed basis. The goal is to provide computing resources that transparently scale to the user community, much as an electrical utility company can deliver power to meet peak demand by accessing energy from other power providers’ plants via a power grid. Oracle Database 10 g further reduces the time, cost, and complexity of database management through the introduction of self-managing features such as the Automated Database Diagnostic Monitor, Automated Shared Memory Tuning, Automated Storage Management, and Automa...
Read more

Insert

Image
      Syntax: INSERT INTO table (column-1, column-2, ... column-n) VALUES (value-1, value-2, ... value-n);                      If we are inserting datas to entire fields then no need to specify the fieldnames.        Example:                                   INSERT INTO EMPLOYEE_DETAILS    VALUES('Akhilash',22,2500,'Developer');          INSERT INTO EMPLOYEE_DETAILS    VALUES('Akhilash',22,250.90,'Developer');     INSERT INTO EMPLOYEE_DETAILS    VALUES('Akhilash',22,250.90,'Developer');     OutPut             ...
Read more

Except

Selecting records from a table, that is not existing in another table.              We can use the keyword EXCEPT example:        select   * from emp1 except select * from emp2   It will display records that exist in emp1 but not in emp2
Read more