Can MySQL reasonably perform queries on billions of rows? — MySQL can ‘handle’ billions of rows. “Reasonably” depends on the queries; let’s see them. Is InnoDB (MySQL 5.5.8) the right choice for multi-billion rows? — 5.7 has some improvements, but 5.5 is pretty good, in spite of being nearly 6 8 years old, and on … Read more
The idea behind partitioning isn’t to use multiple servers but to use multiple tables instead of one table. You can divide a table into many tables so that you can have old data in one sub table and new data in another table. Then the database can optimize queries where you ask for new data … Read more
You’ll want to do something that makes use of List.subList(int, int) views rather than copying each sublist. To do this really easily, use Guava’s Lists.partition(List, int) method: List<Foo> foos = … for (List<Foo> partition : Lists.partition(foos, n)) { // do something with partition } Note that this, like many things, isn’t very efficient with a … Read more
What version of pandas are you using? For me your code works fine (i`m on git master). Another approach could be: In [117]: import pandas In [118]: import random In [119]: df = pandas.DataFrame(np.random.randn(100, 4), columns=list(‘ABCD’)) In [120]: rows = random.sample(df.index, 10) In [121]: df_10 = df.ix[rows] In [122]: df_90 = df.drop(rows) Newer version (from … Read more
Well, lets make your dataset marginally more interesting: val rdd = sc.parallelize(for { x <- 1 to 3 y <- 1 to 2 } yield (x, None), 8) We have six elements: rdd.count Long = 6 no partitioner: rdd.partitioner Option[org.apache.spark.Partitioner] = None and eight partitions: rdd.partitions.length Int = 8 Now lets define small helper to … Read more
Spark >= 2.3.0 SPARK-22614 exposes range partitioning. val partitionedByRange = df.repartitionByRange(42, $”k”) partitionedByRange.explain // == Parsed Logical Plan == // ‘RepartitionByExpression [‘k ASC NULLS FIRST], 42 // +- AnalysisBarrier Project [_1#2 AS k#5, _2#3 AS v#6] // // == Analyzed Logical Plan == // k: string, v: int // RepartitionByExpression [k#5 ASC NULLS FIRST], 42 … Read more
Yes, Zookeeper is required for running Kafka. From the Kafka Getting Started documentation: Step 2: Start the server Kafka uses zookeeper so you need to first start a zookeeper server if you don’t already have one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node zookeeper instance. As to why, … Read more