Spark Streaming - Kafka - java.nio.BufferUnderflowException -


i'm running below error while trying consume message kafka through spark streaming (kafka direct api). used work ok when using spark standalone cluster manager. switched using cloudera 5.7 using yarn manage spark cluster , started see below error.

few details: - spark 1.6.0 - using kafka direct stream api - kafka broker version (0.8.2.1) - kafka version in classpath of yarn executors (0.9) - kafka brokers not managed cloudera

the difference see between using standalone cluster manager , yarn kafka version being used on consumer end. (0.8.2.1 vs 0.9)

trying figure if version mismatch issue ? if indeed case, fix other upgrading kafka brokers 0.9 well. (eventually yes not now)

org.apache.spark.sparkexception: job aborted due stage failure: task 0 in stage 200.0 failed 4 times, recent failure: lost task 0.3 in stage 200.0 (tid 203,..): java.nio.bufferunderflowexception @ java.nio.heapbytebuffer.get(heapbytebuffer.java:151) @ java.nio.bytebuffer.get(bytebuffer.java:715) @ kafka.api.apiutils$.readshortstring(apiutils.scala:40) @ kafka.api.topicdata$.readfrom(fetchresponse.scala:96) @ kafka.api.fetchresponse$$anonfun$4.apply(fetchresponse.scala:170) @ kafka.api.fetchresponse$$anonfun$4.apply(fetchresponse.scala:169) @ scala.collection.traversablelike$$anonfun$flatmap$1.apply(traversablelike.scala:251) @ scala.collection.traversablelike$$anonfun$flatmap$1.apply(traversablelike.scala:251) @ scala.collection.immutable.range.foreach(range.scala:141) @ scala.collection.traversablelike$class.flatmap(traversablelike.scala:251) @ scala.collection.abstracttraversable.flatmap(traversable.scala:105) @ kafka.api.fetchresponse$.readfrom(fetchresponse.scala:169) @ kafka.consumer.simpleconsumer.fetch(simpleconsumer.scala:135) @ org.apache.spark.streaming.kafka.kafkardd$kafkardditerator.fetchbatch(kafkardd.scala:192) @ org.apache.spark.streaming.kafka.kafkardd$kafkardditerator.getnext(kafkardd.scala:208) @ org.apache.spark.util.nextiterator.hasnext(nextiterator.scala:73) @ scala.collection.iterator$$anon$11.hasnext(iterator.scala:327) @ scala.collection.iterator$class.foreach(iterator.scala:727) @ scala.collection.abstractiterator.foreach(iterator.scala:1157) @ scala.collection.generic.growable$class.$plus$plus$eq(growable.scala:48) @ scala.collection.mutable.arraybuffer.$plus$plus$eq(arraybuffer.scala:103) @ scala.collection.mutable.arraybuffer.$plus$plus$eq(arraybuffer.scala:47) @ scala.collection.traversableonce$class.to(traversableonce.scala:273) @ scala.collection.abstractiterator.to(iterator.scala:1157) @ scala.collection.traversableonce$class.tobuffer(traversableonce.scala:265) @ scala.collection.abstractiterator.tobuffer(iterator.scala:1157) @ scala.collection.traversableonce$class.toarray(traversableonce.scala:252) @ scala.collection.abstractiterator.toarray(iterator.scala:1157) @ org.apache.spark.rdd.rdd$$anonfun$tolocaliterator$1$$anonfun$org$apache$spark$rdd$rdd$$anonfun$$collectpartition$1$1.apply(rdd.scala:942) @ org.apache.spark.rdd.rdd$$anonfun$tolocaliterator$1$$anonfun$org$apache$spark$rdd$rdd$$anonfun$$collectpartition$1$1.apply(rdd.scala:942) @ org.apache.spark.sparkcontext$$anonfun$runjob$5.apply(sparkcontext.scala:1869) @ org.apache.spark.sparkcontext$$anonfun$runjob$5.apply(sparkcontext.scala:1869) @ org.apache.spark.scheduler.resulttask.runtask(resulttask.scala:66) @ org.apache.spark.scheduler.task.run(task.scala:89) @ org.apache.spark.executor.executor$taskrunner.run(executor.scala:214) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617) @ java.lang.thread.run(thread.java:745)

driver stacktrace: @ org.apache.spark.scheduler.dagscheduler.org$apache$spark$scheduler$dagscheduler$$failjobandindependentstages(dagscheduler.scala:1431) @ org.apache.spark.scheduler.dagscheduler$$anonfun$abortstage$1.apply(dagscheduler.scala:1419) @ org.apache.spark.scheduler.dagscheduler$$anonfun$abortstage$1.apply(dagscheduler.scala:1418) @ scala.collection.mutable.resizablearray$class.foreach(resizablearray.scala:59)


-

Comments

Post a Comment

Popular posts from this blog

scala - 'wrong top statement declaration' when using slick in IntelliJ -

Image
i'm porting project scala 2.9 2.10, therefore have use slick instead of scalaquery. i'm using slick 2.1.0 since supports ms access. according this tutorial , upgrade guide changed robs object class , added val robs : why error message wrong top statement declaration , how rid of it? edit: i'm new scala... according this question seems can't put val outside of methods or classes, right? code above directly in packge. right approach slick then? should move code in class or trait? according this changed val robs = tablequery[robs] to object robs extends tablequery(new robs(_)) {} . no warnings or errors anymore. :)
Read more

c# - DevExpress.Wpf.Grid.InfiniteGridSizeException was unhandled -

Image
when using devexpress, see error: devexpress.wpf.grid.infinitegridsizeexception unhandled message="by default, infinite grid height not allowed since grid rows rendered , hence grid work slowly. fix issue, should place grid container give finite height grid, or should manually specify grid's height or maxheight. note can avoid exception setting gridcontrol.allowinfinitegridsize static property true, in case grid run slowly." the problem dxgrid has infinite height. to fix, set height non-infinite. snoop absolutely invaluable this: if "height" xaml element infinite (i.e. 0 or nan ), can set using one of following: option 1: height="{binding path=actualheight, relativesource={relativesource mode=findancestor, ancestortype=uielement}}" option 2: verticalalignment="stretch" option 3: height="auto" hint: use verticalalignment="stretch" if child of grid <rowdefinition height=
Read more

PySide and Qt Properties: Connecting signals from Python to QML -

i trying run simple pyside appliccation design made in qml. call function in qml python in separated thread. i have following qml view: import qt 4.7 rectangle { visible: true width: 100; height: 100 color: "lightgrey" text { id: datestr objectname: "datestr" x: 10 y: 10 text: "init text" font.pixelsize: 12 function updatedata(text) { datestr.text = qstr(text) console.log("you said: " + text) } } } here application: some imports.... #!/usr/bin/python import sys import time pyside import qtdeclarative, qtcore pyside.qtgui import qapplication pyside.qtdeclarative import qdeclarativeview the qthread worker.... #worker thread class worker(qtcore.qthread): updateprogress = qtcore.signal(int) def __init__(self, child): qtcore.qthread.__init__(self) self.child = child def run(self):
Read more