java - Creating an RDD after retrieving data from cassandra DB -

-

i'm using cassandra , spark project, wrote retrieve data db:

 results = session.execute("select * foo.test");   arraylist<string> supportlist = new arraylist<string>();  (row row : results) {             supportlist.add(row.getstring("firstcolumn") + "," + row.getstring("secondcolumn")));         }         javardd<string> input = sparkcontext.parallelize(supportlist);         javapairrdd<string, double> tuple = input.maptopair(new pairfunction<string, string, double>() {             public tuple2<string, double> call(string x) {                 string[] parts = x.split(",");                 return new tuple2(parts[0],string.valueof(new random().nextint(30) + 1));             }

it works, want know if there pretty way write above code, want achieve is:

  • in scala can retrieve , fill rdd in way :

    val datardd = sc.cassandratable[tablecolumnnames]("keyspace", "table")

  • how can write same thing in java, without using support list or other "nasty" things.

update

javardd<string> cassandrarowsrdd = javafunctions(javasparkcontext).cassandratable("keyspace", "table")                 .map(new function<cassandrarow, string>() {                     @override                     public string call(cassandrarow cassandrarow) throws exception {                         return cassandrarow.tostring();                     }                 });

i'm getting on row -> public string call(cassandrarow cassandrarow) exception:

exception in thread "main" org.apache.spark.sparkexception: task not serializable     @ org.apache.spark.util.closurecleaner$.ensureserializable(closurecleaner.scala:166)     @ org.apache.spark.util.closurecleaner$.clean(closurecleaner.scala:158)     @ org.apache.spark.sparkcontext.clean(sparkcontext.scala:1623)     @ org.apache.spark.rdd.rdd.map(rdd.scala:286)     @ org.apache.spark.api.java.javarddlike$class.map(javarddlike.scala:89)     @ org.apache.spark.api.java.abstractjavarddlike.map(javarddlike.scala:46)     @ org.sparkexamples.cassandraexample.main.kmeans.executequery(kmeans.java:271)     @ org.sparkexamples.cassandraexample.main.kmeans.main(kmeans.java:67) caused by: java.io.notserializableexception: org.sparkexamples.cassandraexample.main.kmeans serialization stack:     - object not serializable (class: org.sparkexamples.cassandraexample.main.kmeans, value: org.sparkexamples.cassandraexample.main.kmeans@3015db78)     - field (class: org.sparkexamples.cassandraexample.main.kmeans$2, name: this$0, type: class org.sparkexamples.cassandraexample.main.kmeans)     - object (class org.sparkexamples.cassandraexample.main.kmeans$2, org.sparkexamples.cassandraexample.main.kmeans$2@5dbf5634)     - field (class: org.apache.spark.api.java.javapairrdd$$anonfun$toscalafunction$1, name: fun$1, type: interface org.apache.spark.api.java.function.function)     - object (class org.apache.spark.api.java.javapairrdd$$anonfun$toscalafunction$1, <function1>)     @ org.apache.spark.serializer.serializationdebugger$.improveexception(serializationdebugger.scala:38)     @ org.apache.spark.serializer.javaserializationstream.writeobject(javaserializer.scala:47)     @ org.apache.spark.serializer.javaserializerinstance.serialize(javaserializer.scala:80)     @ org.apache.spark.util.closurecleaner$.ensureserializable(closurecleaner.scala:164)     ... 7 more

thanks in advance.

have @ answer: rdd not serializable cassandra/spark connector java api

the problem may class surrounding code block you've shown not serializable.


Comments

Post a Comment

Popular posts from this blog

qt - Using float or double for own QML classes -

-
i've created components , helper methods using c++ , qt used in qml gui. of methods calculate gui elements (size, transformations, scene graph items creation). target arm 32 bit using opengl / eglfs render output. opengl using float , qml using real (defined double ). what should use in c++ do not lose precision? if use qreal ( double ) shouldn't lose precision. if git grep float in qtbase.git , can find out bit more: dist/changes-5.2.0-**************************************************************************** dist/changes-5.2.0-* important behavior changes * dist/changes-5.2.0-**************************************************************************** dist/changes-5.2.0- dist/changes-5.2.0- - qt compiled qreal typedef'ed double on dist/changes-5.2.0: platforms. qreal float on arm chipsets before. guarantees more dist/changes-5.2.0- consistent behavior between platforms qt supports, binary dist/chan...
Read more

Create Outlook appointment via C# .Net -

-
i need create outlook appointment using microsoft.office.interop.outlook , while can work locally on own workstation , works if run web application via server's browser, not work when connect server externally. hence, think might permission issue. i changed application pool identity " localsystem " don't access denied error. unfortunately, doesn't work. the app behaves if appointment created, appointment doesn't appear in outlook. here include file on top of aspx.cs page using outlook = microsoft.office.interop.outlook; here code using pop appointment. outlook.application apptapp = new outlook.application(); outlook.appointmentitem appt = apptapp.createitem(outlook.olitemtype.olappointmentitem) outlook.appointmentitem; appt.subject = txtfirstname.text + " " + txtlastname.text; appt.body = txtcomment.text; appt.alldayevent = false; appt.start = datetime.parse(txtreminderdate.text + " 8:00 am"); appt.end = datetime.p...
Read more

ios - Swift Array Resetting Itself -

-
i have 2 arrays: pointhistory , datehistory . have nstimer , , when timer runs out, action called. in action appending int pointhistory , , current date string datehistory . here timer action: func timeraction() { let steppervalue = int(stepper.value) employee.numberofpoints = steppervalue networking.saveemployee(employee, completion: { (error) -> void in if error != nil { let alert = uialertcontroller(title: "error", message: "there error updating \(self.employee.name)'s points", preferredstyle: .alert) let alertaction = uialertaction(title: "ok", style: .cancel, handler: nil) alert.addaction(alertaction) self.presentviewcontroller(alert, animated: true, completion: nil) } }) var dateformatter = nsdateformatter() dateformatter.dateformat = "mm-dd-yyyy" dateformatter.timezone = nstimezone.localtimezone() var date = nsdate() var ...
Read more