linux
linux show file and dir sizes
For current dir for file and dirs in current path, with max depth 1 or just simply(h, d, 1 args merged as one)
spark
extract json value directly in pyspark
Say a spark dataframe has a column named json_str_col which contains json format strings, and the json format string have the format {“key1” : “some value”} we can directly extract key1 ‘s values as a new column with the following.
spark
release cached dataframe in pyspark
to release all cached dataframes use the following reference: https://sparkbyexamples.com/spark/spark-drop-dataframe-from-cache/
python
pyenv prerequisite packages to install
pyenv: https://github.com/pyenv/pyenv link the packages to install: https://github.com/pyenv/pyenv/wiki#suggested-build-environment
error fix
fix blank plot when using plotly in jupyter lab
this helped: https://github.com/plotly/plotly.py/issues/2508#issuecomment-907338746
error fix
pylance stuck on “Searching for source files”
while using vscode, I noticed that pylance wasn’t running even after forcing pylance server restart. I checked the output logs and it was stuck at this line: solution add pyrightconfig.json on project root dir. Here is a link to github Read more…
deep learning
paper review: “Graph Attention Networks”
arxiv: https://arxiv.org/abs/1710.10903 key points introduce “graph attention network(GAT)” which consists of “graph attention layers” which incorporate the “self-attention” idea from transformers to graph neural network the graph attention layers work by calculating weights of a node’s neighbors from the features Read more…
error fix
fix “Certificate verification failed” error in apt update of docker container
Background While working on a raw docker base image that didn’t even have basic tools installed, I tried to call apt update but the following error came up This error occurred even after setting “http_proxy” and “https_proxy” environment variables. Solution Read more…
linux
fixing “Could not handshake: An unexpected TLS packet was received” error while apt update in docker container behind corporate proxy
Background While trying to build a docker image from a very raw apache spark base image, since it didn’t have any basic packages such as vim, ssh, wget, etc, I entered a running container of this image and typed apt Read more…