amazon s3 - Easiest way to persist Cassandra data to S3 using Spark -

-

i trying figure out how best store , retrieve data, s3 cassandra, using spark: have log data store in cassandra. run spark using dse perform analysis of data, , works beautifully. log data grows daily, , need 2 weeks worth in cassandra @ given time. still need store older logs somewhere @ least 6 months, , after research, s3 glaciar looks promising solution. i'd use spark, run daily job finds logs day 15, deletes them cassandra, , sends them s3. problem this: can't seem settle on right format save cassandra rows file, such can 1 day potentially load file spark, , run analysis, if have to. want run analysis in spark 1 day, not persist data cassandra. json seems obvious solution, there other format not considering? should use spark sql? advice appreciated before commit 1 format or another.

apache spark designed kind of use case. storage format columnar databases. provides column compression , indexing.

it becoming de facto standard. many big data platforms adopting or @ least providing support it. can query efficiently directly in s3 using sparksql, impala or apache drill. can run emr jobs against it.

to write data parquet using spark, use dataframe.saveasparquetfile.

depending on specific requirements may end not needing separate cassandra instance.

you may find this post interesting


Comments

Post a Comment

Popular posts from this blog

qt - Using float or double for own QML classes -

-
i've created components , helper methods using c++ , qt used in qml gui. of methods calculate gui elements (size, transformations, scene graph items creation). target arm 32 bit using opengl / eglfs render output. opengl using float , qml using real (defined double ). what should use in c++ do not lose precision? if use qreal ( double ) shouldn't lose precision. if git grep float in qtbase.git , can find out bit more: dist/changes-5.2.0-**************************************************************************** dist/changes-5.2.0-* important behavior changes * dist/changes-5.2.0-**************************************************************************** dist/changes-5.2.0- dist/changes-5.2.0- - qt compiled qreal typedef'ed double on dist/changes-5.2.0: platforms. qreal float on arm chipsets before. guarantees more dist/changes-5.2.0- consistent behavior between platforms qt supports, binary dist/chan...
Read more

Create Outlook appointment via C# .Net -

-
i need create outlook appointment using microsoft.office.interop.outlook , while can work locally on own workstation , works if run web application via server's browser, not work when connect server externally. hence, think might permission issue. i changed application pool identity " localsystem " don't access denied error. unfortunately, doesn't work. the app behaves if appointment created, appointment doesn't appear in outlook. here include file on top of aspx.cs page using outlook = microsoft.office.interop.outlook; here code using pop appointment. outlook.application apptapp = new outlook.application(); outlook.appointmentitem appt = apptapp.createitem(outlook.olitemtype.olappointmentitem) outlook.appointmentitem; appt.subject = txtfirstname.text + " " + txtlastname.text; appt.body = txtcomment.text; appt.alldayevent = false; appt.start = datetime.parse(txtreminderdate.text + " 8:00 am"); appt.end = datetime.p...
Read more

ios - Swift Array Resetting Itself -

-
i have 2 arrays: pointhistory , datehistory . have nstimer , , when timer runs out, action called. in action appending int pointhistory , , current date string datehistory . here timer action: func timeraction() { let steppervalue = int(stepper.value) employee.numberofpoints = steppervalue networking.saveemployee(employee, completion: { (error) -> void in if error != nil { let alert = uialertcontroller(title: "error", message: "there error updating \(self.employee.name)'s points", preferredstyle: .alert) let alertaction = uialertaction(title: "ok", style: .cancel, handler: nil) alert.addaction(alertaction) self.presentviewcontroller(alert, animated: true, completion: nil) } }) var dateformatter = nsdateformatter() dateformatter.dateformat = "mm-dd-yyyy" dateformatter.timezone = nstimezone.localtimezone() var date = nsdate() var ...
Read more