Convert clojure vector to flambo sql row -

-

i'm working on developing function convert vector sql row further convert data-frame , save table using sqlcontext in apache spark. i'm developing in clojure , got lost along way. thought of implementing solution thus:

  1. for each rdd (vector) convert rows
  2. convert rows data frame
  3. save data frame table
  4. use sqlcontext query particular information in table

  5. and how convert result query into rdd again further analysis.

    (defn assign-ecom    []    (let [rdd-fields (-> (:rdd @transformed-rdd)                  (f/map #(sql/row->vec %))                   f/collect)]      (clojure.pprint/pprint rdd-fields)))

i'm using flambo v0.60 api function abstracting apache-spark functions, welcome suggestion how go solving problem.

here's link flambo row -> vec docs:

flambo documentation:

i assume have spark-context (sc) , sql-context (sql-ctx). first lets import stuff we'll need:

(import org.apache.spark.sql.rowfactory) (import org.apache.spark.sql.types.structtype) (import org.apache.spark.sql.types.structfield) (import org.apache.spark.sql.types.metadata) (import org.apache.spark.sql.types.datatypes)

  1. for each rdd (vector) convert rows

    ;; vector row conversion (defn vec->row [v]    (rowfactory/create (into-array object v)))  ;; example data (def rows (-> (f/parallelize sc [["foo" 1] ["bar" 2]])               (f/map vec->row)))

  2. convert rows data frame

    ;; define schema (def schema   (structtype.    (into-array structfield      [(structfield. "k" (datatypes/stringtype) false (metadata/empty))       (structfield. "v" (datatypes/integertype) false (metadata/empty))])))  ;; create data frame (def df (.createdataframe sql-ctx rows schema))  ;; see if works (.show df)

  3. save data frame table

    (.registertemptable df "df")

  4. use sqlcontext query particular information in table

    (def df-keys (.sql sql-ctx "select upper(k) k df")) ;; check results (.show df-keys)

  5. and how convert result query into rdd again further analysis.

    (.tojavardd df-keys)

    or if want vectors:

    (f/map (.tojavardd df-keys) sql/row->vec)

Comments

Post a Comment

Popular posts from this blog

qt - Using float or double for own QML classes -

-
i've created components , helper methods using c++ , qt used in qml gui. of methods calculate gui elements (size, transformations, scene graph items creation). target arm 32 bit using opengl / eglfs render output. opengl using float , qml using real (defined double ). what should use in c++ do not lose precision? if use qreal ( double ) shouldn't lose precision. if git grep float in qtbase.git , can find out bit more: dist/changes-5.2.0-**************************************************************************** dist/changes-5.2.0-* important behavior changes * dist/changes-5.2.0-**************************************************************************** dist/changes-5.2.0- dist/changes-5.2.0- - qt compiled qreal typedef'ed double on dist/changes-5.2.0: platforms. qreal float on arm chipsets before. guarantees more dist/changes-5.2.0- consistent behavior between platforms qt supports, binary dist/chan...
Read more

Create Outlook appointment via C# .Net -

-
i need create outlook appointment using microsoft.office.interop.outlook , while can work locally on own workstation , works if run web application via server's browser, not work when connect server externally. hence, think might permission issue. i changed application pool identity " localsystem " don't access denied error. unfortunately, doesn't work. the app behaves if appointment created, appointment doesn't appear in outlook. here include file on top of aspx.cs page using outlook = microsoft.office.interop.outlook; here code using pop appointment. outlook.application apptapp = new outlook.application(); outlook.appointmentitem appt = apptapp.createitem(outlook.olitemtype.olappointmentitem) outlook.appointmentitem; appt.subject = txtfirstname.text + " " + txtlastname.text; appt.body = txtcomment.text; appt.alldayevent = false; appt.start = datetime.parse(txtreminderdate.text + " 8:00 am"); appt.end = datetime.p...
Read more

ios - Swift Array Resetting Itself -

-
i have 2 arrays: pointhistory , datehistory . have nstimer , , when timer runs out, action called. in action appending int pointhistory , , current date string datehistory . here timer action: func timeraction() { let steppervalue = int(stepper.value) employee.numberofpoints = steppervalue networking.saveemployee(employee, completion: { (error) -> void in if error != nil { let alert = uialertcontroller(title: "error", message: "there error updating \(self.employee.name)'s points", preferredstyle: .alert) let alertaction = uialertaction(title: "ok", style: .cancel, handler: nil) alert.addaction(alertaction) self.presentviewcontroller(alert, animated: true, completion: nil) } }) var dateformatter = nsdateformatter() dateformatter.dateformat = "mm-dd-yyyy" dateformatter.timezone = nstimezone.localtimezone() var date = nsdate() var ...
Read more