Why is it that my work is doing so slowly?
Let me describe it, and then tell me if it's like you: you're doing a four-line task on a gigabyte of data with two innocent connections, and it takes a hell of an hour to do. Or another option: you have an hour-long job that's done smoothly until the 1149/1150 problem is stuck, and two hours later you decide to kill it, because you don't know if it's a spark bug, or some big data god who's mad at you!
Then you say, "Hmm, maybe my Spark cluster is too small, let me raise the processor and memory." Then... The same. Amazon must be laughing right now. So it has to be a million-dollar question.
You're looking at the only Spark optimization course on the network. With the techniques you're learning here, you'll save time, money, energy and get rid of headaches.
Let's fix it.
With this course, we cut the weeds under the root. We delve deeply into Spark and understand why it takes so long to work before we touch any code or, worse, spend money on computing. And then we bring the guns. You'll learn 20 degrees of techniques and optimization strategies. Each of them individually can give at least a double increase in the productivity of your work, and I show it on camera.
What awaits you:
- You'll understand Spark's internal device to explain whether you're writing good code or not.
- You'll be able to predict in advance whether it will take a long time
- You'll read query plans and DAG while you're completing assignments to see if you're doing something wrong.
- You optimize DataFrame conversions far beyond the standard Spark auto-optimizer.
- You'll be quick user-processing with efficient RDDs, as opposed to the S'L.
- You diagnose hang-ups, stages and tasks
- You'll discover and correct data skews
- In addition, you'll fix a few memory failures along the way
