本篇內(nèi)容介紹了“mapPartitions的簡(jiǎn)單介紹及使用方法”的有關(guān)知識(shí)，在實(shí)際案例的操作過程中，不少人都會(huì)遇到這樣的困境，接下來就讓小編帶領(lǐng)大家學(xué)習(xí)一下如何處理這些情況吧！希望大家仔細(xì)閱讀，能夠?qū)W有所成！
在成都做網(wǎng)站、成都網(wǎng)站制作、成都外貿(mào)網(wǎng)站建設(shè)中從網(wǎng)站色彩、結(jié)構(gòu)布局、欄目設(shè)置、關(guān)鍵詞群組等細(xì)微處著手，突出企業(yè)的產(chǎn)品/服務(wù)/品牌，幫助企業(yè)鎖定精準(zhǔn)用戶，提高在線咨詢和轉(zhuǎn)化，使成都網(wǎng)站營銷成為有效果、有回報(bào)的無錫營銷推廣。創(chuàng)新互聯(lián)建站專業(yè)成都網(wǎng)站建設(shè)10多年了，客戶滿意度97.8%，歡迎成都創(chuàng)新互聯(lián)客戶聯(lián)系。
1. mappartition簡(jiǎn)介
重慶分公司
重慶分公司
/** * Return a new RDD by applying a function to each partition of this RDD. * * `preservesPartitioning` indicates whether the input function preserves the partitioner, which * should be `false` unless this is a pair RDD and the input function doesn't modify the keys. */ def mapPartitions[U: ClassTag]( f: Iterator[T] => Iterator[U], preservesPartitioning: Boolean = false): RDD[U] = withScope { val cleanedF = sc.clean(f) new MapPartitionsRDD( this, (_: TaskContext, _: Int, iter: Iterator[T]) => cleanedF(iter), preservesPartitioning) } /** * Return a new RDD by applying a function to all elements of this RDD. */ def map[U: ClassTag](f: T => U): RDD[U] = withScope { val cleanF = sc.clean(f) new MapPartitionsRDD[U, T](this, (_, _, iter) => iter.map(cleanF)) } val a = sc.parallelize(1 to 20, 2)def mapTerFunc(a : Int) : Int = {a*3}val mapResult = a.map(mapTerFunc)println(mapResult.collect().mkString(","))
3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57,60
3. mappartitions低效用法
val a = sc.parallelize(1 to 20, 2)def terFunc(iter: Iterator[Int]) : Iterator[Int] = {var res = List[Int]()while (iter.hasNext){val cur = iter.next;res.::= (cur*3) ;}res.iterator}val result = a.mapPartitions(terFunc)println(result.collect().mkString(","))
30,27,24,21,18,15,12,9,6,3,60,57,54,51,48,45,42,39,36,33
4. mappartitions高效用法
class CustomIterator(iter: Iterator[Int]) extends Iterator[Int] {
def hasNext : Boolean = {
iter.hasNext
}
def next : Int= {
val cur = iter.next
cur*3
}
}
val result = a.mapPartitions(v => new CustomIterator(v))
println(result.collect().mkString(","))
3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57,60
“mapPartitions的簡(jiǎn)單介紹及使用方法”的內(nèi)容就介紹到這里了，感謝大家的閱讀。如果想了解更多行業(yè)相關(guān)的知識(shí)可以關(guān)注創(chuàng)新互聯(lián)網(wǎng)站，小編將為大家輸出更多高質(zhì)量的實(shí)用文章！
