Mastering Big Data Analytics with PySpark, Published by Packt

License: MIT License

Dockerfile 0.13% Jupyter Notebook 65.40% HTML 33.21% Python 1.26%

mastering-big-data-analytics-with-pyspark's Introduction

Mastering Big Data Analytics with PySpark [Video]

This is the code repository for Mastering Big Data Analytics with PySpark [Video], published by Packt. It contains all the supporting project files necessary to work through the video course from start to finish. Authored by: Danny Meijer

About the Video Course

PySpark helps you perform data analysis at-scale; it enables you to build more scalable analyses and pipelines. This course starts by introducing you to PySpark's potential for performing effective analyses of large datasets. You'll learn how to interact with Spark from Python and connect Jupyter to Spark to provide rich data visualizations. After that, you'll delve into various Spark components and its architecture.

You'll learn to work with Apache Spark and perform ML tasks more smoothly than before. Gathering and querying data using Spark SQL, to overcome challenges involved in reading it. You'll use the DataFrame API to operate with Spark MLlib and learn about the Pipeline API. Finally, we provide tips and tricks for deploying your code and performance tuning.

By the end of this course, you will not only be able to perform efficient data analytics but will have also learned to use PySpark to easily analyze large datasets at-scale in your organization.

What You Will Learn

Gain a solid knowledge of vital Data Analytics concepts via practical use cases
Create elegant data visualizations using Jupyter
Run, process, and analyze large chunks of datasets using PySpark
Utilize Spark SQL to easily load big data into DataFrames
Create fast and scalable Machine Learning applications using MLlib with Spark
Perform exploratory Data Analysis in a scalable way
Achieve scalable, high-throughput and fault-tolerant processing of data streams using Spark Streaming

Instructions and Navigation

Assumed Knowledge

This course will greatly appeal to data science enthusiasts, data scientists, or anyone who is familiar with Machine Learning concepts and wants to scale out his/her work to work with big data.

If you find it difficult to analyze large datasets that keep growing, then this course is the perfect guide for you!

A working knowledge of Python assumed.

Technical Requirements

Minimum Hardware Requirements

For successful completion of this course, students will require the computer systems with at least the following:

OS: Windows, Mac, or Linux Processor: Any processor from the last few years Memory: 2GB RAM Storage: 300MB for the Integrated Development Environment (IDE) and 1GB for cache

Recommended Hardware Requirements

For an optimal experience with hands-on labs and other practical activities, we recommend the following configuration:

OS: Windows, Mac, or Linux Processor: Core i5 or better (or AMD equivalent) Memory: 8GB RAM or better Storage: 2GB free for build caches and dependencies

Software Requirements

Operating system: Windows, Mac, or Linux Docker

Follow the instructions below to download the data belonging to the course as well as

setting up your interactive development environment.

Downloading Data for this Course

Once you have cloned this repository locally, simply navigate to the folder you have stored the repo in and run: python download_data.py

This will populate the data-sets folder in your repo with a number of data sets that will be used throughout the course.

Docker Image Bundled with the Course

About

The Docker Image bundled with this course (see Dockerfile) is based on the pyspark-notebook, distributed and maintained by Jupyter

Github link Original copyright (c) Jupyter Development Team. Distributed under the terms of the Modified BSD License.

This Course's Docker image extends the pyspark-notebook with the following additions:

enables Jupyter Lab by default
exposes correct ports for JupyterLab and SparkUI
sets numerous default settings to improve Quality of Life for the user
installs numerous add-ons (such as pyspark-stubs and blackcellmagic) using jupyter_contrib_nbextensions

Instructions for use

There are 2 ways to access the Docker container in this course:

Through the bundled run_me.py script (recommended to use)
Through the Docker CLI (only for advanced users)

Using the bundled script to run the container

The easiest way to run the container that belongs to this course is by running python run_me.py from the course's repository. This will automatically build the Docker image, set up the Docker container, download the data, and set up the necessary volume mounts.

Using Docker CLI

If you rather start the Docker container manually, use the following instructions:

Download the data
```
python download_data.py
```

Build the image

docker build --rm -f "Dockerfile" -t mastering_pyspark_ml:latest .

Run the image Ensure that you replace /path/to/mastering_pyspark_ml/repo/ in the following command, and run it in a terminal or command prompt:

docker run  -v /path/to/mastering_pyspark_ml/repo/:/home/jovyan/ --rm -d -p 8888:8888 -p 4040:4040 --name mastering_pyspark_ml mastering_pyspark_ml .

Open Jupyter lab once Docker image is running Navigate to http://localhost:8888/lab

To Stop the Docker Image

Once you are ready to shutdown the Docker container, you can use the following command:

docker stop mastering_pyspark_ml

mastering-big-data-analytics-with-pyspark's People

Contributors

Stargazers

Watchers

Forkers

sotojcr vamosraghava ngtantai bielolopez charles-bate krdinesh19 kostat-data-science-research flyinacres nguyendo24 swagatd entzyeung buratina mehmettbaki jinyuantian tedd2 opentrainingcamp varunmuriyanat rgupta77 sandeepvvs007 sinha-ujjawal alaadiab7 alhaol luongvy nurulakbartanjung hashphoenix srmds rossanomarcos vidhurraj147 vmdhhh dipeshsapkota101 cgiannotta javonnii kannan2109 pradeep699 vpatrikis rongshengzhang anlimaye spark-bigdata-analytics-engine cmatheusia edmund-george agredac twrnrg skarangi-personal sketchjar ctrivino1 adamzas rajaramkuberan isamaljawarneh fatna-sys rajasahe kausikvaleti sou-786 123prostab roodee benjaminbenteke shubhamkrp therealhaseeb prohith995 arifmudi apizarroe parth-github caroljiao seanreed1111 diteric qichun-dai emreaytkn ajay291491 takenev59 salama4ai yuchild wichodrivera luisaarellano javaslee sayantanr ashryverrr muhammeedsari brijesh-deb micseb asanalacli priyatransbit rohitsingh0001 llenroc navin534 bailang1208 genostack divanov137 marcelomgarcia mathmachado khileshchauhan irononet krishnamurtyp 153079019shariq datastudysquad cs-suraj ocipoc hadibabaei94 yowens2 teshomegit carbirbal sachin3366

mastering-big-data-analytics-with-pyspark's Issues

can not download the data sources

I am able to run the download_data.py but it only created a data folder without downloading any source files.

Trace back when running run_me.py

I am running an Ubuntu 20.04 LTS host on VMWARE. Check the Readme and requirements. All the modules have been installed. Python version (from conda - 4.9.2) is 3.8.5

Trying to execute the run_me.py script, but getting the following errors

(base) phil@ubuntu:~/Project/Mastering-Big-Data-Analytics-with-PySpark-master (3)/Mastering-Big-Data-Analytics-with-PySpark-master$ python run_me.py
INFO CourseHandler Welcome to 'Mastering Big Data Analytics with Pyspark' by Danny Meijer [email protected]
INFO CourseHandler Course Version: 20200420
INFO CourseHandler Course Name: Mastering Big Data Analytics with Pyspark
INFO CourseHandler Container Name: mastering-pyspark-ml
INFO CourseHandler Downloading the data
INFO CourseHandler Processing Sentiment140TrainingData
INFO CourseHandler Directory "sentiment-140-training-data" already exists
INFO CourseHandler Skipping "trainingandtestdata.zip"
INFO CourseHandler Processing MovieLens-Small
INFO CourseHandler Directory "ml-latest-small" already exists
INFO CourseHandler Skipping "ml-latest-small.zip"
INFO CourseHandler Processing MovieLens
INFO CourseHandler Directory "ml-latest" already exists
INFO CourseHandler Skipping "ml-latest.zip"
INFO CourseHandler Connecting to Docker API
Traceback (most recent call last):
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/phil/Downloads/yes/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/phil/Downloads/yes/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/phil/Downloads/yes/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/phil/Downloads/yes/lib/python3.8/http/client.py", line 1010, in _send_output
self.send(msg)
File "/home/phil/Downloads/yes/lib/python3.8/http/client.py", line 950, in send
self.connect()
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/urllib3/util/retry.py", line 531, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/phil/Downloads/yes/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/phil/Downloads/yes/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/phil/Downloads/yes/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/phil/Downloads/yes/lib/python3.8/http/client.py", line 1010, in _send_output
self.send(msg)
File "/home/phil/Downloads/yes/lib/python3.8/http/client.py", line 950, in send
self.connect()
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/docker/transport/unixconn.py", line 30, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run_me.py", line 324, in
course = Course()
File "run_me.py", line 114, in init
self._client()
File "run_me.py", line 152, in _client
self.client = docker.from_env()
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/docker/client.py", line 96, in from_env
return cls(
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/docker/client.py", line 45, in init
self.api = APIClient(*args, **kwargs)
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/docker/api/client.py", line 197, in init
self._version = self._retrieve_server_version()
File "/home/phil/Downloads/yes/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Missing ipynb-File in Section 8

I have downloaded the current version of the github repo and I noticed that in Section 8 in the twitter_app folder the file twitter_app.ipynb is missing.

I am working with your online course at packt publishing.

Regards,

Vinz Frauchiger

Dataset

Running container inside a vm

If you are running the container inside a vm, you should change line 290:
port_map = {"{}/tcp".format(p): ("127.0.0.1", p) for p in self.ports}
to
port_map = {"{}/tcp".format(p): ("0.0.0.0", p) for p in self.ports}
otherwise you cant connect from outside your vm

Invalid syntax at line 135 in run_me.py

Mastering-Big-Data-Analytics-with-PySpark. on  master [!] via 🐍 v2.7.16 took 1m39s
❯ python run_me.py
  File "run_me.py", line 135
    **{
     ^

Environment

Mastering-Big-Data-Analytics-with-PySpark. on  master [!] via 🐍 v2.7.16
❯ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.2
BuildVersion:	19C57

Mastering-Big-Data-Analytics-with-PySpark. on  master [!] via 🐍 v2.7.16
❯ python --version
Python 2.7.16

Building docker image via run_me.py fails

INFO CourseHandler Welcome to 'Mastering Big Data Analytics with Pyspark' by Danny Meijer [email protected]
INFO CourseHandler Course Version: 20200420
INFO CourseHandler Course Name: Mastering Big Data Analytics with Pyspark
INFO CourseHandler Container Name: mastering-pyspark-ml
INFO CourseHandler Downloading the data
INFO CourseHandler Processing Sentiment140TrainingData
INFO CourseHandler Directory "sentiment-140-training-data" already exists
INFO CourseHandler Skipping "trainingandtestdata.zip"
INFO CourseHandler Processing MovieLens-Small
INFO CourseHandler Directory "ml-latest-small" already exists
INFO CourseHandler Skipping "ml-latest-small.zip"
INFO CourseHandler Processing MovieLens
INFO CourseHandler Directory "ml-latest" already exists
INFO CourseHandler Skipping "ml-latest.zip"
INFO CourseHandler Connecting to Docker API
INFO CourseHandler Checking if Docker image is already set-up
WARNING CourseHandler Docker image has not been built yet
INFO CourseHandler Building Docker image
INFO docker.api.build Initiating (this might take a few moments)
Traceback (most recent call last):
File "run_me.py", line 325, in
course = Course()
File "run_me.py", line 117, in init
self._image()
File "run_me.py", line 256, in _image
self.build_image()
File "run_me.py", line 194, in build_image
rm=True,
File "C:\Users\Hansrajjadhav\AppData\Local\Programs\Python\Python37\lib\site-packages\docker\api\build.py", line 261, in build
self._set_auth_headers(headers)
File "C:\Users\Hansrajjadhav\AppData\Local\Programs\Python\Python37\lib\site-packages\docker\api\build.py", line 308, in _set_auth_headers
auth_data = self.auth_configs.get_all_credentials()
File "C:\Users\Hansrajjadhav\AppData\Local\Programs\Python\Python37\lib\site-packages\docker\auth.py", line 302, in get_all_credentials
for k in store.list().keys():
File "C:\Users\Hansrajjadhav\AppData\Local\Programs\Python\Python37\lib\site-packages\docker\credentials\store.py", line 72, in list
return json.loads(data.decode('utf-8'))
File "C:\Users\Hansrajjadhav\AppData\Local\Programs\Python\Python37\lib\json_init.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\Hansrajjadhav\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Hansrajjadhav\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

run_me.py: FileNotFoundError: [Errno 2] No such file or directory

Hi I was trying to run python run_me.py command as suggested in the README, however ended up having a bit long, hard to understand error messages. I use pipenv and created a pyhton3.6.5 environment for this course.

Please help me decipher the following issues:

(ozkans) ozkans@OZKANs-MacBook-Pro Mastering-Big-Data-Analytics-with-PySpark % python --version
Python 3.6.5
(ozkans) ozkans@OZKANs-MacBook-Pro Mastering-Big-Data-Analytics-with-PySpark % python run_me.py
INFO CourseHandler Welcome to 'Mastering Big Data Analytics with Pyspark' by Danny Meijer [email protected]
INFO CourseHandler Course Version: 20200420
INFO CourseHandler Course Name: Mastering Big Data Analytics with Pyspark
INFO CourseHandler Container Name: mastering-pyspark-ml
INFO CourseHandler Downloading the data
INFO CourseHandler Processing Sentiment140TrainingData
INFO CourseHandler Directory "sentiment-140-training-data" already exists
INFO CourseHandler Skipping "trainingandtestdata.zip"
INFO CourseHandler Processing MovieLens-Small
INFO CourseHandler Directory "ml-latest-small" already exists
INFO CourseHandler Skipping "ml-latest-small.zip"
INFO CourseHandler Processing MovieLens
INFO CourseHandler Directory "ml-latest" already exists
INFO CourseHandler Skipping "ml-latest.zip"
INFO CourseHandler Connecting to Docker API
Traceback (most recent call last):
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/urllib3/util/retry.py", line 410, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/docker/api/client.py", line 205, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/docker/api/client.py", line 228, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/requests/sessions.py", line 543, in get
return self.request('GET', url, **kwargs)
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run_me.py", line 329, in
course = Course()
File "run_me.py", line 113, in init
self._client()
File "run_me.py", line 153, in _client
self.client = docker.from_env()
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/docker/client.py", line 85, in from_env
timeout=timeout, version=version, **kwargs_from_env(**kwargs)
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/docker/client.py", line 40, in init
self.api = APIClient(*args, **kwargs)
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/docker/api/client.py", line 188, in init
self._version = self._retrieve_server_version()
File "/Users/ozkans/.local/share/virtualenvs/ozkans-QsBVdfJX/lib/python3.6/site-packages/docker/api/client.py", line 213, in _retrieve_server_version
'Error while fetching server API version: {0}'.format(e)
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

No module named 'docker'

Hello,
when i try to run python run_me.py I get the following error:
ModuleNotFoundError: No module named 'docker'

Can you please help?
Thank you
Riccardo

'NoneType' object has no attribute 'id'

The run_my.py script seems to be running fine and almost finishes when I get this error.

INFO docker.api.build 58.14% | 26 Packages, 7 Downloading, 19 Extracted INFO docker.api.build 94.66% | 26 Packages, 7 Downloading, 19 Extracted INFO docker.api.build 94.66% | 26 Packages, 7 Downloading, 19 Extracted INFO docker.api.build Image was built successfully Traceback (most recent call last): File "run_me.py", line 324, in <module> course = Course() File "run_me.py", line 115, in __init__ self._image() File "run_me.py", line 255, in _image self.build_image() File "run_me.py", line 244, in build_image logger.info("Image ID: %s", self.image.id) AttributeError: 'NoneType' object has no attribute 'id'

run_me.py script runs without error but fails to initialise containers.

run_me.py script runs without error but fails to initialise containers. Because of this lab environment fails to load.

Notebook for 4.5 is Missing

Hi, I'm looking for the video you referenced in your Packt training series that has a notebook for section 4-5 with ML code. It's not in your github. Is that somewhere online that I can retrieve it?

packtpublishing / mastering-big-data-analytics-with-pyspark Goto Github PK

mastering-big-data-analytics-with-pyspark's Introduction

Mastering Big Data Analytics with PySpark [Video]

About the Video Course

What You Will Learn

Instructions and Navigation

Assumed Knowledge

Technical Requirements

Minimum Hardware Requirements

Recommended Hardware Requirements

Software Requirements

Follow the instructions below to download the data belonging to the course as well as

Downloading Data for this Course

Docker Image Bundled with the Course

About

Instructions for use

Using the bundled script to run the container

Using Docker CLI

To Stop the Docker Image

Related Products

mastering-big-data-analytics-with-pyspark's People

Contributors

Stargazers

Watchers

Forkers

mastering-big-data-analytics-with-pyspark's Issues

Recommend Projects

Recommend Topics

Recommend Org