[Git] Git submodule basic commands

  • Clone a repo tree
git clone --recurse-submodules remote_url
  • Link a submodule
git submodule add submodule_remote_url local_linking_path
# Then commit & push to the main repo
  • Pull from a submodule
# Pull from specific branch
git pull --recurse-submodules my-branch
# Then commit & push to the main repo
  • Update a submodule
cd submodule_path
# Do the commits and push normally, the changes will be pushed to submodule repo

cd main
# Then do a commit and push to the main module
  • Reset submodule head to currently stored reference
git submodule update --init

[K8s] Command cheatsheet

  • View config
kubectl config view
  • Show cluster info
kubectl cluster-info

# Example output
Kubernetes control plane is running at https://10.0.0.9:8443
KubeDNS is running at https://10.0.0.9:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
  • Get component info
kubectl get xxx -o wide -A
kubectl describe xxx

# Use -l $LABEL to filter using labels
# USe --show-labels to show labels

# xxx can be nodes, pods, deployments,
# events, services, replicaset, etc.
# To specify a specific object: xxx/name. For example: deloy/my-deploy

# For full list and the short names, do 
kubectl api-resources
  • Create/Edit/Delete any object
# Create
kubectl create -f config.yaml

# Edit. Can be used to rollout a new version when editing deployments
kubectl edit xxx/$NAME

# Delete
kubectl delete xxx/$NAME
kubectl delete -f config.yaml
  • Create a deployment which manages a Pod
kubectl create deployment hello-node --image=k8s.gcr.io/echoserver:1.4
  • Expose a deployment as a service
# 8080 here is the container port
# NodePort exposes the ports on each node running related pods
kubectl expose deployment/hello-node --type="NodePort" --port 8080
  • Scale a deployment
kubectl scale deployments/kubernetes-bootcamp --replicas=4
  • Update a deployment
kubectl set image deployments/kubernetes-bootcamp kubernetes-bootcamp=jocatalin/kubernetes-bootcamp:v2 --record=true

# Check the rollout status
kubectl rollout status deployments/kubernetes-bootcamp

# Undo the rollout
# This rollback to previous replicaset
kubectl rollout undo deployments/kubernetes-bootcamp
  • Undo a rollout
# Find the replicaset you want to restore
# (default 10 history replicasets are saved)
kubectl get rs -o wide
kubectl rollout history $DEPLOY

# Get the revision number of the desired replicaset
kubectl describe rs kubernetes-bootcamp-fb5c67579

# Rollback to the specific replicaset
kubectl rollout undo deployments/kubernetes-bootcamp --to-revision=2
  • Logging
kubectl logs $POD_NAME
  • Execute a command on a container in a Pod
kubectl exec $POD_NAME -- $COMMAND


# Start a bash session. If no container specified, kubectl.kubernetes.io/default-container will be used
kubectl exec -it $POD_NAME -c $CONTAINER_NAME -- bash
  • Access a Pod’s port locally w/o services (for debugging)
# Method 1. Port-forwarding
kubectl port-forward POD_NAME HOST_PORT:POD_PORT
# Then you can access the endpoint at 
# localhost:HOST_PORT

# Method 2. Proxy
kubectl proxy
# Then you can access the endpoint at
# http://localhost:8001/api/v1/namespaces/default/pods/$POD_NAME:$POD_PORT/proxy/
  • Labeling
# Attach a new label
kubectl label pods $POD_NAME version=v1

# View labels of a Pod
kubectl describe pods $POD_NAME
  • Secrets
    Ways to use secrets:
    • As container envs (secretKeyRef)
    • As volumes mounted to containers
    • Save to a docker image and provide access inside the cluster
# Create a secret
kubectl create secret generic $SECRET [--from-file file_name] [--from-literal]

# Show a secret
kubectl get secret/xxx -o json
  • ConfigMap
kubectl create configmap $CONFIGMAP [--from-file] [--from-literal]
  • Drain a Node
# Drain a Node to move all Pods to other Nodes
kubectl drain $NODE

# Undo draining
kubectl uncordon $NODE to revert
  • Clean up
kubectl delete service hello-node
kubectl delete deployment hello-node
minikube stop
minikube delete

[K8s] Basic concept notes

Main components

Source
  • Each Node (machine) includes a kubelet and many Pods
  • Kubelets communicate with the Control Plane (actually a special node) and the corresponding Pods inside the Node
  • Control Plane can be split into multiple machines
  • Each Pod serves as an application and has many containers which are tightly coupled
  • Containers inside the same Pod share the same networking resource
  • kube-apiserver: Exposes the Kubernetes API
  • etcd: Key value store
  • kube-scheduler: Schedule Nodes to run Pods
  • kube-controller-manager: Many components deal with monitoring, task running, and account access control

History of container runtime

Source
  • From the oldest method (top) to newest approach (bottom)
  • CRI is an interface designed for kubelet to manipulate containers
  • Any CRI-compliant container runtime can do the job: Docker Engine, containerd, CRI-O, …
  • OCI runtime: runC, runhcs, …

K8s entity concepts

Source
  • A Deployment checks on the health of your Pod and restarts the Pod’s container if it terminates. Deployments are the recommended way to manage the creation and scaling of Pods
  • Deployment configuration file works very similar to docker-compose.yml
  • Deployment controller automatically search for a suitable node where an instance of the application could be run
  • If the Node hosting an instance goes down or is deleted, the Deployment controller replaces the instance with an instance on another Node in the cluster
  • A Service is an exposed Deployment, which encapsulates a set of Pods called ReplicaSet
  • Secrets and ConfigMap are extra resources we can mount into containers
  • Volumes types:
    • emptyDir: Storage lives with a Pod
    • hostPath: Storage lives with a Node
    • NFS and other cloud provider type
  • Storage
    • Storage Classes: Templates to create volumes
    • Persistent Volume Claims: Create (or use existing) PV by specifying SC, size, read/write type etc. We can then attach a PVC into a Pod config
    • Persistent Volumes: Either created manually (static) or automatically (dynamic) when no any PV match the claim
  • Resource
    • Request: minimum requirement
    • Quota: maximum limit

Common tools

  • cri-o: A container runtime
  • kubelet: Starts pods and containers
  • kubectl: The command line util to talk to your cluster.
  • kubeadm: The command to bootstrap the cluster (control planes etc.)
  • minikube: A tool to quickly setup a single-node cluster inside a VM
  • Kops: Setup AWS clusters easily

Networking

  • Each Node has its own node IP (as normal machines) which is mainly accessed by the control plane node
  • Each Pod has its own IP, where the Node will forward the request to
  • Pods inside each Node, or across Nodes, can see each other using Pod IPs, but not recommended to do so. Exposing the endpoint as a service is preferred
  • Containers inside the same Pod can refer to each other using localhost:port, this is like running up docker-compose inside a “pod machine”, but the container network is configured as host
  • Services have Cluster IPs and domain name, for intra-cluster access
  • Port definitions:
    • port: For intra-cluster access
    • nodePort: Exposed port on the Node
    • targetPort: Container port
  • Ingress along with ingress controller act like an Nginx load balancer. The DNS record should point to this load balancer IP to serve services across multiple Nodes

[Linux] Vim pasting causes extra spaces

When you pasting text from other sources into Vim using mouse right-click, the format may be messed up due to auto-indenting. Use the following command to enable pasting mode:

:set paste

After the pasting is done, revert to normal mode:

:set nopaste

[SQL] Common syntax

  • DISTINCT
SELECT DISTINCT state FROM airports;
  • COUNT / SUM / AVG / MAX / MIN
SELECT COUNT(DISTINCT state) AS state_count
  FROM airports;

SELECT s.sname, s.age
  FROM sailors AS s
 WHERE s.age =
       (SELECT MAX(s2.age)
          FROM sailors AS s2);

/* NULL is counted in COUNT, but ignored in other aggregate operators */
  • WHERE & Conditions
SELECT *
  FROM airports
 WHERE state = 'NY' 
   AND (city = 'Perry' OR city != 'Middletown');
  • LIKE
SELECT *
  FROM airports
 WHERE city LIKE '%er%';
/* % is 0 or more arbitrary char, _ is one arbitrary char */
  • BETWEEN
SELECT *
  FROM airports
 WHERE lat BETWEEN 30 AND 30.5;
  • ORDER BY & ASC & DSC & LIMIT
SELECT *
FROM airports
ORDER BY lat, city DESC
LIMIT 20;
  • UNION / INTERSECT / EXCEPT
SELECT r.sid
  FROM boats AS b JOIN reserves AS r
    ON r.bid = b.bid AND b.color = 'red'
INTERSECT
SELECT r.sid
  FROM boats AS b JOIN reserves AS r
    ON r.bid = b.bid AND b.color = 'green';

/* INTERSECT (and EXCEPT) can be written as follow */
/* Method No.1 */
SELECT s.sid
  FROM sailors AS s
       JOIN boats AS b1 JOIN reserves AS r1
       JOIN boats AS b2 JOIN reserves AS r2
    ON s.sid = r1.sid AND r1.bid = b1.bid
   AND s.sid = r2.sid AND r2.bid = b2.bid
   AND (b1.color = 'red' AND b2.color = 'green');

/* Method No.2 */
SELECT s.name
  FROM sailors AS s
       JOIN boats AS b
       JOIN reserves AS r
    ON s.sid = r.sid AND r.bid = b.bid AND b.color = 'red'
   AND s.sid IN 
       (SELECT r2.sid
          FROM boats AS b2 JOIN reserves AS r2
            ON r2.bid = b2.bid
           AND b2.color = 'green');
  • IN / EXISTS / UNIQUE
SELECT s.sname
  FROM sailors AS s
 WHERE EXISTS
       (SELECT *
          FROM reserves AS r
         WHERE r.bid = 103
           AND s.sid = r.sid);

SELECT s.sname
  FROM sailors AS s
 WHERE s.sid IN (12, 13, 14, 15);
  • ALL / ANY
SELECT *
  FROM sailors AS s
 WHERE s.rating > ALL 
       (SELECT s2.rating
          FROM sailors AS s2
         WHERE s2.sname = 'Horatio');
  • GROUP BY / HAVING
SELECT s.rating, MIN(s.age) AS min_age
  FROM sailors AS s
 WHERE s.age >= 18
GROUP BY s.rating
HAVING COUNT(*) > 1;

/* 
- HAVING clause must be aggregation among each group 
  or attributes listed in GROUP BY.
- Aggregation operations cannot be nested like MIN(AVG(...)).
*/
  • INSERT INTO
INSERT INTO albums (title, release_date, recording_date)
VALUES ('Charcoal Lane', '1990-01-01 01:01:01.00000', '1990-01-01 01:01:01.00000'),
       ('The New Danger', '2008-01-01 01:01:01.00000', '1990-01-01 01:01:01.00000');
  • UPDATE
UPDATE staff
   SET salary = salary * 1.3, vacation = vacation + 5
 WHERE title = 'Manager';
  • DELETE FROM
DELETE FROM albums
 WHERE title = 'Charcoal Lane';
  • LEFT JOIN / RIGHT JOIN / FULL JOIN (outer joins)
/* Preserve all entries in either side or both sides after join */
SELECT s.sid, r.bid
  FROM reserves AS r 
       RIGHT JOIN sailors AS s
       on r.sid = s.sid
          AND s.rating > 3;
  • CHECK & CONSTRAINT
CREATE TABLE staff (
    PRIMARY KEY (staff_num),
    staff_num      INT(5)       NOT NULL,
    first_name     VARCHAR(100) NOT NULL,
    pens_in_drawer INT(2)       NOT NULL,
                   CONSTRAINT pens_in_drawer_range
                   CHECK(pens_in_drawer BETWEEN 1 AND 99)
);
  • ASSERTION
CREATE ASSERTION small_club
 CHECK ((SELECT COUNT(s.sid) FROM sailors AS s)
       + (SELECT COUNT(b.bid) FROM boats AS b) < 100);
  • TRIGGER
CREATE TRIGGER young_sailor_update
AFTER INSERT ON sailors
REFERENCING NEW TABLE new_sailors
FOR EACH STATEMENT
INSERT INTO young_sailors (sid, name, age, rating)
     SELECT sid, name, age, rating
       FROM new_sailors as n
      WHERE n.age <= 18;
  • Division example

A / B means “selecting the keys of A which contain all the values of B”.
For example, find sailors who’ve reserved all boats.

sailors: (*sid, sname, rating, age)
boats: (*bid, bname, color)
reserves: (*sid, *bid, day)

SELECT s.name
  FROM sailors AS s
 WHERE NOT EXIST
       ((SELECT b.bid
           FROM boats AS b)
        EXCEPT
        (SELECT r.bid
           FROM reserves AS r
          WHERE r.sid = s.sid)
       );

* We can actually write EXCEPT using IN as described previously.

[Python] Quick sort

def quick_sort(nums, left=None, right=None):
    def _partition(nums, left, right, pivot_idx):
        pivot = nums[pivot_idx]
        nums[pivot_idx], nums[right] = nums[right], nums[pivot_idx]
        tmp_idx = left
        for idx in range(left, right):
            if nums[idx] <= pivot:
                nums[tmp_idx], nums[idx] = nums[idx], nums[tmp_idx]
                tmp_idx += 1
        nums[tmp_idx], nums[right] = nums[right], nums[tmp_idx]
        return tmp_idx

    if left is None:
        left = 0
    if right is None:
        right = len(nums)-1
    if left >= right:
        return

    new_pivot_idx = _partition(nums, left, right, left)
    quick_sort(nums, left, new_pivot_idx-1)
    quick_sort(nums, new_pivot_idx+1, right)


# inputs = input()
# inputs = [int(c) for c in inputs.split()]
inputs = [3, 5, 1, 2, 6, 4, 7]
quick_sort(inputs)
print(inputs)

[Agile] Kanban & Scrum brief comparison

ScrumKanban
CadenceRegular fixed length sprintsContinuous flow
Release methodApproved by the product ownerContinuous delivery
RolesProduct owner, scrum master, development teamNo existing roles
Key metricsVelocityCycle time
ChangeTeams should strive to not make changes to the sprint forecast during the sprint.Change can happen at any time

Reference

[Linux] NTFS、 FAT32、exFAT檔案系統比較

檔案系統FAT32NTFSextFAT
作業系統Win, Linux, MacWin
Linux, Mac: Read-only
Win, Linux, Mac
單檔大小 4GB 16TB 64ZB
速度SlowestFastestMiddle
穩定性GoodBestWorst

[Unix] Unix, Linux, macOS之關係

Unix-like作業系統關係圖

  • Unix-like作業系統,包含了Unix的衍生系統以及與Unix類似的系統
    • Unix的直系衍生系統:FreeBSD, macOS…
    • Unix類似的系統(獨立開發):Linux, Minix…
  • 因此Linux及macOS都是Unix-like作業系統,但Linux是獨立於Unix開發,而macOS是基於BSD開發的系統

*資料來源

[Git] Show logs for specific branch

If you have master merged into your current branch, your git log will be messy because it also shows the commits from the master.

If you want to show your branch logs, use the following command:

git log master..

or

git log master..mybranch

* git log a..b actually means “includes commits that are (backward) reachable from b but exclude those that are reachable from a”