不会耗尽内存的Spark打印数据帧

link之家

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

不会耗尽内存的Spark打印数据帧

Dataset<Row> df = ...

df.show()

df.show(rowCount, false)

df.foreachPartition(iter -> {
    while(iter.hasNext()){
       System.out.println(rowIter.next().mkString(",");)
});

df.write.mode("overwrite").saveAsTable("database.tableName")

df.write.csv("/your/location/data.csv")
 df.write.json("/your/location/data.json")

long count = df.count();
long inc = count / 10;
for (long i = 0; i < count; i += inc) {
  Dataset<Row> filteredDf =
      df.where("id>=" + i + " AND id<" + (i + inc));
  List<Row> rows = filteredDf.collectAsList();
  for (Row r : rows) {
    System.out.printf("%d: %s\n", r.getAs(0), r.getString(1));
}

package net.jgp.books.sparkWithJava.ch20.lab900_splitting_dataframe;
import java.util.ArrayList;
import java.util.List;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.RowFactory;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.DataTypes;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
 * Splitting a dataframe to bring it back to the driver for local
 * processing.
 * @author jgp
public class SplittingDataframeApp {
   * main() is your entry point to the application.
   * @param args
  public static void main(String[] args) {
    SplittingDataframeApp app = new SplittingDataframeApp();
    app.start();
   * The processing code.
  private void start() {
    // Creates a session on a local master
    SparkSession spark = SparkSession.builder()
        .appName("Splitting a dataframe to collect it")
        .master("local")
        .getOrCreate();
    Dataset<Row> df = createRandomDataframe(spark);
    df = df.cache();
    df.show();
    long count = df.count();
    long inc = count / 10;
    for (long i = 0; i < count; i += inc) {
      Dataset<Row> filteredDf =
          df.where("id>=" + i + " AND id<" + (i + inc));
      List<Row> rows = filteredDf.collectAsList();
      for (Row r : rows) {
        System.out.printf("%d: %s\n", r.getAs(0), r.getString(1));
  private static Dataset<Row> createRandomDataframe(SparkSession spark) {
    StructType schema = DataTypes.createStructType(new StructField[] {
        DataTypes.createStructField(
            "id",
            DataTypes.IntegerType,
            false),
        DataTypes.createStructField(
            "value",
            DataTypes.StringType,
            false) });
    List<Row> rows = new ArrayList<Row>();
    for (int i = 0; i < 100; i++) {

推荐文章

活泼的泡面 · SQL Server 2008备份数据库失败，拒绝访问的原因-阿里云开发者社区

1 月前

刚毅的眼镜 · edge浏览器使用IE模式进行调试_iechooser

2 周前

任性的冰淇淋 · 怎样使用IETester调试页面-百度经验

2 周前

眉毛粗的毛衣 · AttributeError: module 'matplotlib' has no attribute 'rcParams_Fix for matplotlib rcParams error_ -

1 周前

精明的猴子 · 如何应对网页视频课程进度条禁止拖动？ - wzzkaifa

1 周前

痴情的雪糕 · Power shell的数学运算_powershell 幂运算-CSDN博客

5 月前

高大的泡面 · 菜单概述 - WPF .NET Framework | Microsoft Learn

1 年前

儒雅的保温杯 · 【问题排查】Spring Cloud Gateway 返回 403 状态码 - 简书

2 年前

博学的双杠 · Oracle 使用SQL*Plus连接数据库 - 授客 - 博客园

2 年前

风流的冰淇淋 · Matlab 编程基础：脚本与实时脚本的创建与调试 - 知乎

2 年前

不会耗尽内存的Spark打印数据帧

2 个回答