這篇文章主要介紹eclipse中如何運行spark機器學(xué)習(xí)代碼，文中介紹的非常詳細(xì)，具有一定的參考價值，感興趣的小伙伴們一定要看完！
成都創(chuàng)新互聯(lián)主營欽南網(wǎng)站建設(shè)的網(wǎng)絡(luò)公司,主營網(wǎng)站建設(shè)方案,成都app軟件開發(fā)公司,欽南h5重慶小程序開發(fā)搭建,欽南網(wǎng)站營銷推廣歡迎欽南等地區(qū)企業(yè)咨詢
直接在eclipse運行，不需要hadoop，不需要搭建spark，只需要pom.xml中的依賴完整
重慶分公司
重慶分公司
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.mllib.classification.LogisticRegressionWithSGD
import org.apache.spark.mllib.feature.HashingTF
import org.apache.spark.mllib.regression.LabeledPoint
object MLlib {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName(s"Book example: Scala").setMaster("local[2]")
val sc = new SparkContext(conf)
// Load 2 types of emails from text files: spam and ham (non-spam).
// Each line has text from one email.
val spam = sc.textFile("file:/Users/xxx/Documents/hadoopTools/scala/eclipse/Eclipse.app/Contents/MacOS/workspace/spark_ml/src/main/resources/files/spam.txt")
val ham = sc.textFile("file:/Users/xxx/Documents/hadoopTools/scala/eclipse/Eclipse.app/Contents/MacOS/workspace/spark_ml/src/main/resources/files/ham.txt")
// val abc=sc.parallelize(seq, 2)
// Create a HashingTF instance to map email text to vectors of 100 features.
val tf = new HashingTF(numFeatures = 100)
// Each email is split into words, and each word is mapped to one feature.
val spamFeatures = spam.map(email => tf.transform(email.split(" ")))
val hamFeatures = ham.map(email => tf.transform(email.split(" ")))
// Create LabeledPoint datasets for positive (spam) and negative (ham) examples.
val positiveExamples = spamFeatures.map(features => LabeledPoint(1, features))
val negativeExamples = hamFeatures.map(features => LabeledPoint(0, features))
val trainingData = positiveExamples ++ negativeExamples
trainingData.cache() // Cache data since Logistic Regression is an iterative algorithm.
// Create a Logistic Regression learner which uses the LBFGS optimizer.
val lrLearner = new LogisticRegressionWithSGD()
// Run the actual learning algorithm on the training data.
val model = lrLearner.run(trainingData)
// Test on a positive example (spam) and a negative one (ham).
// First apply the same HashingTF feature transformation used on the training data.
val posTestExample = tf.transform("O M G GET cheap stuff by sending money to ...".split(" "))
val negTestExample = tf.transform("Hi Dad, I started studying Spark the other ...".split(" "))
// Now use the learned model to predict spam/ham for new emails.
println(s"Prediction for positive test example: ${model.predict(posTestExample)}")
println(s"Prediction for negative test example: ${model.predict(negTestExample)}")
sc.stop()
}
}sc.textFile里的參數(shù)是文件在本地的絕對路徑。
setMaster("local[2]") 表示是本地運行，只使用兩個核
HashingTF 用來從文檔中創(chuàng)建詞條目的頻率特征向量，這里設(shè)置維度為100.
TF-IDF(Term frequency-inverse document frequency ) 是文本挖掘中一種廣泛使用的特征向量化方法。TF-IDF反映了語料中單詞對文檔的重要程度。假設(shè)單詞用t表示，文檔用d表示，語料用D表示，那么文檔頻度DF(t, D)是包含單詞t的文檔數(shù)。如果我們只是使用詞頻度量重要性，就會很容易過分強調(diào)重負(fù)次數(shù)多但攜帶信息少的單詞，例如：”a”, “the”以及”of”。如果某個單詞在整個語料庫中高頻出現(xiàn)，意味著它沒有攜帶專門針對某特殊文檔的信息。逆文檔頻度(IDF)是單詞攜帶信息量的數(shù)值度量。
pom.xml
