apache spark - How to partition data by multiple fields? -

-

lets have record 4 identifier variables: var1, var2, var3, var4 , additional variable: var5.

i want perform reduce operation on records have same value in either 1 of identifier fields. i'm trying think how can implement kind of solution minimal amount of shuffling.

is there option tell spark put records have @ least 1 match in identifier variables on same partition? know there option of custom partitioner i'm not sure if possible implement in way support use case.

well, quite lot depends on structure of data in general , how priori knowledge have.

in worst case scenario, when data relatively dense , uniformly distributed below , perform one-off analysis, way achieve goal seems to put 1 partition.

[1 0 1 0] [0 1 0 1] [1 0 0 1]

obviously not useful approach. 1 thing can try instead analyze @ least subset of data insight structure , try use knowledge build custom partitioner assures relatively low traffic , reasonable distribution on cluster @ same time.

as general framework try 1 of these:

hashing

  • choose number of buckets
  • for each row create binary vector of length equal number of buckets
    • for each feature in row
      • hash feature bucket
        • if hash(bucket) == 0 flip 1
        • otherwise nothing
  • sum computed vectors summary statistics
  • use optimization technique of choice create function hash partition

frequent itemsets

  • use 1 of algorithms apriori, closed-patterns, max-patterns on sample of data, fp-growth
  • compute distribution of itemsets on sample
  • use optimization compute hash above

both solutions computationally intensive , require quite lot of work implement not worth fuss ad hoc analytics if have reusable pipeline may worth trying.


Comments

Post a Comment

Popular posts from this blog

qt - Using float or double for own QML classes -

-
i've created components , helper methods using c++ , qt used in qml gui. of methods calculate gui elements (size, transformations, scene graph items creation). target arm 32 bit using opengl / eglfs render output. opengl using float , qml using real (defined double ). what should use in c++ do not lose precision? if use qreal ( double ) shouldn't lose precision. if git grep float in qtbase.git , can find out bit more: dist/changes-5.2.0-**************************************************************************** dist/changes-5.2.0-* important behavior changes * dist/changes-5.2.0-**************************************************************************** dist/changes-5.2.0- dist/changes-5.2.0- - qt compiled qreal typedef'ed double on dist/changes-5.2.0: platforms. qreal float on arm chipsets before. guarantees more dist/changes-5.2.0- consistent behavior between platforms qt supports, binary dist/chan...
Read more

Create Outlook appointment via C# .Net -

-
i need create outlook appointment using microsoft.office.interop.outlook , while can work locally on own workstation , works if run web application via server's browser, not work when connect server externally. hence, think might permission issue. i changed application pool identity " localsystem " don't access denied error. unfortunately, doesn't work. the app behaves if appointment created, appointment doesn't appear in outlook. here include file on top of aspx.cs page using outlook = microsoft.office.interop.outlook; here code using pop appointment. outlook.application apptapp = new outlook.application(); outlook.appointmentitem appt = apptapp.createitem(outlook.olitemtype.olappointmentitem) outlook.appointmentitem; appt.subject = txtfirstname.text + " " + txtlastname.text; appt.body = txtcomment.text; appt.alldayevent = false; appt.start = datetime.parse(txtreminderdate.text + " 8:00 am"); appt.end = datetime.p...
Read more

ios - Swift Array Resetting Itself -

-
i have 2 arrays: pointhistory , datehistory . have nstimer , , when timer runs out, action called. in action appending int pointhistory , , current date string datehistory . here timer action: func timeraction() { let steppervalue = int(stepper.value) employee.numberofpoints = steppervalue networking.saveemployee(employee, completion: { (error) -> void in if error != nil { let alert = uialertcontroller(title: "error", message: "there error updating \(self.employee.name)'s points", preferredstyle: .alert) let alertaction = uialertaction(title: "ok", style: .cancel, handler: nil) alert.addaction(alertaction) self.presentviewcontroller(alert, animated: true, completion: nil) } }) var dateformatter = nsdateformatter() dateformatter.dateformat = "mm-dd-yyyy" dateformatter.timezone = nstimezone.localtimezone() var date = nsdate() var ...
Read more