public class PartitionedBroadcast<T extends CacheBlock> extends Object implements Serializable
Broadcast<PartitionedBlock> into Broadcast<PartitionedBlock>[],
 we got java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE issue.
 Despite various jiras, this issue still showed up in Spark 2.1.| Constructor and Description | 
|---|
| PartitionedBroadcast() | 
| PartitionedBroadcast(org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[] broadcasts,
                    DataCharacteristics dc) | 
| Modifier and Type | Method and Description | 
|---|---|
| static int | computeBlocksPerPartition(long[] dims,
                         int blen) | 
| static int | computeBlocksPerPartition(long rlen,
                         long clen,
                         long blen) | 
| void | destroy()This method cleanups all underlying broadcasts of a partitioned broadcast,
 by forward the calls to SparkExecutionContext.cleanupBroadcastVariable. | 
| T | getBlock(int[] ix) | 
| T | getBlock(int rowIndex,
        int colIndex) | 
| org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[] | getBroadcasts() | 
| DataCharacteristics | getDataCharacteristics() | 
| long | getNumCols() | 
| int | getNumColumnBlocks() | 
| int | getNumRowBlocks() | 
| long | getNumRows() | 
| T | slice(long rl,
     long ru,
     long cl,
     long cu,
     T block)Utility for slice operations over partitioned matrices, where the index range can cover
 multiple blocks. | 
public PartitionedBroadcast()
public PartitionedBroadcast(org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[] broadcasts, DataCharacteristics dc)
public org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[] getBroadcasts()
public long getNumRows()
public long getNumCols()
public int getNumRowBlocks()
public int getNumColumnBlocks()
public DataCharacteristics getDataCharacteristics()
public static int computeBlocksPerPartition(long rlen,
                                            long clen,
                                            long blen)
public static int computeBlocksPerPartition(long[] dims,
                                            int blen)
public T getBlock(int rowIndex, int colIndex)
public T getBlock(int[] ix)
public T slice(long rl, long ru, long cl, long cu, T block)
rl - row lower boundru - row upper boundcl - column lower boundcu - column upper boundblock - block objectpublic void destroy()
Copyright © 2020 The Apache Software Foundation. All rights reserved.