linux
linux show file and dir sizes
For current dir for file and dirs in current path, with max depth 1 or just simply(h, d, 1 args merged as one)
For current dir for file and dirs in current path, with max depth 1 or just simply(h, d, 1 args merged as one)
Say a spark dataframe has a column named json_str_col which contains json format strings, and the json format string have the format {“key1” : “some value”} we can directly extract key1 ‘s values as a new column with the following.
to release all cached dataframes use the following reference: https://sparkbyexamples.com/spark/spark-drop-dataframe-from-cache/
pyenv: https://github.com/pyenv/pyenv link the packages to install: https://github.com/pyenv/pyenv/wiki#suggested-build-environment
this helped: https://github.com/plotly/plotly.py/issues/2508#issuecomment-907338746
while using vscode, I noticed that pylance wasn’t running even after forcing pylance server restart. I checked the output logs and it was stuck at this line: solution add pyrightconfig.json on project root dir. Here is a link to github Read more…
arxiv: https://arxiv.org/abs/1710.10903 key points introduce “graph attention network(GAT)” which consists of “graph attention layers” which incorporate the “self-attention” idea from transformers to graph neural network the graph attention layers work by calculating weights of a node’s neighbors from the features Read more…
Background While working on a raw docker base image that didn’t even have basic tools installed, I tried to call apt update but the following error came up This error occurred even after setting “http_proxy” and “https_proxy” environment variables. Solution Read more…
Background While trying to build a docker image from a very raw apache spark base image, since it didn’t have any basic packages such as vim, ssh, wget, etc, I entered a running container of this image and typed apt Read more…