[Python] Multiprocessing task with tqdm

Python / 2020 年 10 月 18 日

We can use tqdm and multiprocessing together by utilizing imap.

import multiprocessing as mp
from tqdm import tqdm

def parallel_work(args):
    my_arg_a= args['my_arg_a']
    my_arg_b= args['my_arg_b']
    result = do_things(my_arg_a, my_arg_b)
    return result

if __name__ == '__main__':
    # Build a list with objects containing desired arguments
    parallel_work_args = [{'my_arg_a': 'a', 'my_arg_b': 'b'}, ...]  

    # If you encounter deadlock issue, try using the 'spawn' method for children forking
    # with mp.get_context("spawn").Pool(mp.cpu_count()) as pool:
    with mp.Pool(mp.cpu_count()) as pool:
        # Parallel work each arg object
        # list() and total are required 
        results = list(tqdm(pool.imap(parallel_work, parallel_work_args), total=len(parallel_work_args)))

[Linux] Screen enable mouse scrolling

Linux / 2020 年 8 月 8 日

To enable mouse scrolling for screen sessions, just put this line into your ~/.screenrc file:

termcapinfo xterm* ti@:te@

To test if it works, you must re-attach your sessions.

[Git] Resolve conflicts for git stash pop

Git / 2020 年 7 月 30 日

Assume you have conflicts after executing git stash pop:

  1. Resolve the conflicts manually by yourself
  2. Use git reset to mark conflicts as resolved
  3. Drop the stash using git stash drop because git doesn’t do it if conflicts occur

[Git] Cut commits onto other branches using git rebase –onto

Git / 2022 年 6 月 1 日

Assume we have following branches:

We want to cut commit_3 and commit_4 on branch_2, onto master (commit_2). We can use the commands:

git checkout branch_2
git rebase --onto b111dc9 34e3359
(git rebase --onto new_base old_base)

The result looks like this:

If we want the commits but not to modify the incoming branch (branch_2 in the above example), we can use cherry-pick :

git checkout master
git cherry-pick 34e3359..321b7c0
(git cherry-pick start_commit_not_included..end_commit_included)

[Git] Pretty tree log

Git / 2020 年 7 月 15 日

This is an easy way to show a pretty git log using alias.

  • Command: 
git log --oneline --decorate --all --graph
  • Add to alias:
git config --global alias.tree "log --oneline --decorate --all --graph"

Then, use git tree to show the tree view:

[JS] Regex related functions usage

JavaScript / 2022 年 5 月 15 日

String.prototype.match()

  • Result format depends on whether the regex has ‘g’ flag.
  • Use for a single match or global matches without group information.
let x = '123abcdef123';
x.match(/(123)/);
// [full_matched, 
//  group1, group2, ..., groupN, 
//  index: index_of_first_match, 
//  input: original_input_string, 
//  groups: named_groups] 

// ["123", 
//  "123", 
//  index: 0, 
//  input: "123abcdef123", 
//  groups: undefined] 

x.match(/(123)/g); 
// NO GROUPS IN RESULTS!!! 
// [first_match, second_match, ..., Nth match] 
// ["123", "123"]

String.prototype.matchAll()

  • Use for global matches where group results are needed.
let x = '123abcdef123'; 
[...x.matchAll(/(123)/g)]; 
// [[first_full_matched, 
//  group1, group2, ..., groupN, 
//  index: index_of_first_match, 
//  input: original_input_string, 
//  groups: named_groups], 
//  [second_full_matched, ...], ...]

RegExp.prototype.exec()

  • Similar use case as String.prototype.match(), but when a sticky flag is used, the regex becomes stateful, calling exec() multiple times on the same string will continue the matching. If global matching is needed, String.prototype.matchAll is preferred for its cleaner code.

RegExp.prototype.test()

String.prototype.search()

  • Both can be used for testing if a string satisfies a regex.

[Git] 使用rebase整理merge commits

Git / 2021 年 12 月 17 日
註解 2020-05-27 112920

使用指令:

git rebase -i 61d3628 -r
註解 2020-05-27 113038

onto為rebase指定的commit，此例為61d3628 。
直接編輯各commit的順序及是否生成merge commit，即可隨心所欲整理merge commits.
下圖為將b2的merge拉直的結果

註解 2020-05-27 113304

[Python] import 機制要點

Python / 2021 年 12 月 17 日
  • https://docs.python.org/3/reference/import.html#package-relative-imports
  • script和module不同，script通常在package外執行
  • package內部，相鄰的module需使用relative import或是absolute import。例如:
package/A.py:
import package.B
from . import B
import package.B.func
from .B import func
# 若只使用”from B import func”會導致”找不到模組”
  • 執行script時，該script的所在位置會被加入path中，因此同階級的script之間是能直接互相import的。

[Linux] Useful SSH tunnel commands

Linux / 2021 年 12 月 17 日
  • 開啟本機的8888 port，通過此port的請求會全部交給remote-host處理(藉由兩者的ssh連線)，此時再把本機的proxy設定成SOCKS localhost:8888，可以達到完全的proxy效果。
ssh -D 8888 -N user@remote-host
  • 開啟本機的8888 port，通過此port的請求會全部通過remote-host到達target-host:80
ssh -N -L 8888:target-host:80 user@remote-host

[Python] Gunicorn

Python / 2021 年 12 月 17 日

Basic usage

gunicorn -k gevent -t 300 -w 4 -b 0.0.0.0:9500 main:app

—Gevent: asynchronous request handling

—Timeout: 300 sec

# of worker processes: 4

Deploy with nginx

/etc/systemd/system/myapi.service

[Unit]
Description=Gunicorn instance for flask API
Requires=myapi.socket
After=network.target

[Service]
User=hubert
Group=hubert
WorkingDirectory=/home/hubert/projectdir
ExecStart=/home/hubert/miniconda3/bin/gunicorn -k gevent -t 300 -w 2 myapp:app
ExecReload=/bin/kill -s HUP $MAINPID
KillMode=mixed
TimeoutStopSec=5
PrivateTmp=true

[Install]
WantedBy=multi-user.target

/etc/systemd/system/myapi.socket

[Unit]
Description=Gunicorn socket for my API.

[Socket]
ListenStream=/run/myapi.sock
# Our service won't need permissions for the socket, since it
# inherits the file descriptor by socket activation
# only the nginx daemon will need access to the socket
User=nginx
# Optionally restrict the socket permissions even more.
# Mode=600

[Install]
WantedBy=sockets.target

nginx site conf file

server {
        listen       9800;
        location / {
            proxy_pass http://unix:/run/myapi.sock;
        }
    }