In my environment (using docker and the image sequenceiq/spark:1.1.0-ubuntu), I ran in to this. If you look at the pyspark shell script, you’ll see that you need a few things added to your PYTHONPATH: export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH That worked in ipython for me. Update: as noted in the comments, the name of the py4j … Read more
Jupyter notebook has a default memory limit size. You can try to increase the memory limit by following the steps: 1) Generate Config file using command: jupyter notebook –generate-config 2) Open jupyter_notebook_config.py file situated inside ‘jupyter’ folder and edit the following property: NotebookApp.max_buffer_size = your desired value Remember to remove the ‘#’ before the property … Read more
You should import the getpass module, then call getpass.getpass. import getpass password = getpass.getpass() Note that in old versions, the field where you enter the password may not appear in the IPython Notebook, but instead in your terminal/command prompt window. In the latest versions of Jupyter Notebook, however, the prompt will appear in the Notebook … Read more
It works for me os.environ[“DJANGO_ALLOW_ASYNC_UNSAFE”] = “true” BTW, I start my notebook using the command ./manage.py shell_plus –notebook
Is this what you want? $ ipython notebook –no-browser Edit Now you should use instead $ jupyter notebook –no-browser Since ipython notebook is deprecated and will be removed in future versions. You likely want to use jupyter notebook in the future
This is a long-running missing feature in jupyter notebooks. I use a near-identical setup: my notebook runs inside a tmux session in a remote server, and I use it locally with ssh tunneling. Before doing any work, I run the following snippet in the first cell: import sys import logging nblog = open(“nb.log”, “a+”) sys.stdout.echo … Read more
If you want your IPython history in a plain-text file, you can export it yourself. %history -g -f filename
When I’m working with python, I usually have two terminal windows open – one with IPython, and the other with a fairly customized Vim. Two good resources: http://blog.dispatched.ch/2009/05/24/vim-as-python-ide/ http://dancingpenguinsoflight.com/2009/02/python-and-vim-make-your-own-ide/ Though it sounds like what you want is IPython’s magic function %ed/%edit: An example of what you can do: In [72]: %ed IPython will make a … Read more
pd.DataFrame.loc can take one or two indexers. For the rest of the post, I’ll represent the first indexer as i and the second indexer as j. If only one indexer is provided, it applies to the index of the dataframe and the missing indexer is assumed to represent all columns. So the following two examples … Read more
I believe the pylab magic was removed when they transitioned from IPython to a more general Jupyter notebook. Try: %matplotlib inline Also when you get a message like: “matplotlib.axes._subplots.AxesSubplot at 0xebf8b70”. That’s just IPython displaying the object. You need to specify IPython display it. Hence the matplotlib inline magic.