Kafka Get Topic Offset

./bin/kafka-run-class.sh kafka.tools.GetOffsetShell –broker-list node1.kafka.es.ops.fraaws.tripws.com:9092 –topic cloud-harbor-registry

Setuid

File modes The setuid and setgid bits are normally represented as the values 4 for setuid and 2 for setgid in the high-order octal digit of the file mode. For example, 6711 has both the setuid and setgid bits (4 + 2 = 6) set, and also the file read/write/executable for the owner (7), and executable by the group (first 1) and others (second 1). Most implementations have a symbolic representation of these bits; in the previous example, this could be u=rwx,go=x,ug+s.

Typically, chmod does not have a recursive mode restricted to directories, so modifying an existing directory tree must be done manually, with a command such as find /path/to/directory -type d -exec chmod g+s ‘{}’ ‘'.

Effects The setuid and setgid flags have different effects, depending on whether they are applied to a file, to a directory or binary executable or non binary executable file. The setuid and setgid flags have an effect only on binary executable files and not on scripts (e.g., Bash, Perl, Python).[3]

When set on an executable file When the setuid or setgid attributes are set on an executable file, then any users able to execute the file will automatically execute the file with the privileges of the file’s owner (commonly root) and/or the file’s group, depending upon the flags set.[2] This allows the system designer to permit trusted programs to be run which a user would otherwise not be allowed to execute. These may not always be obvious. For example, the ping command may need access to networking privileges that a normal user cannot access; therefore it may be given the setuid flag to ensure that a user who needs to ping another system can do so, even if their own account does not have the required privilege for sending packets.

Tcp Error Ack

如果 tcp 数据里面的 ack 是错误的，会发生什么呢？

Golang Broken Pipe

https://gosamples.dev/broken-pipe/

package main

import (
	"errors"
	"log"
	"net"
	"os"
	"syscall"
	"time"
)

func server() {
	listener, err := net.Listen("tcp", ":8080")
	if err != nil {
		log.Fatal(err)
	}

	defer listener.Close()

	conn, err := listener.Accept()
	if err != nil {
		log.Fatal("server", err)
		os.Exit(1)
	}
	data := make([]byte, 1)
	if _, err := conn.Read(data); err != nil {
		log.Fatal("server", err)
	}

	conn.Close()
}

func client() {
	conn, err := net.Dial("tcp", "localhost:8080")
	if err != nil {
		log.Fatal("client", err)
	}

	// write to make the connection closed on the server side
	if n, err := conn.Write([]byte("a")); err != nil {
		log.Printf("client: %v", err)
	} else {
		log.Printf("client: %d", n)
	}

	time.Sleep(1 * time.Second)

	// write to generate an RST packet
	if _, err := conn.Write([]byte("b")); err != nil {
		log.Printf("client: %v", err)
	}

	time.Sleep(1 * time.Second)

	// write to generate the broken pipe error
	if _, err := conn.Write([]byte("c")); err != nil {
		log.Printf("client: %v", err)
		if errors.Is(err, syscall.EPIPE) {
			log.Print("This is broken pipe error")
		}
	}
}

func main() {
	go server()

	time.Sleep(3 * time.Second) // wait for server to run

	client()
}

Prlimit

prlimit 可以通过系统调用更改一下进程的 limit 值。

/proc/pid/limit 查看进程 limit

Soft Hard Limit In Linux

https://docs.oracle.com/cd/E19455-01/805-7229/6j6q8svfe/index.html

You can set both soft and hard limits. The system will not allow a user to exceed his or her hard limit. However, a system administrator may set a soft limit (sometimes referred to as a quota) which can be temporarily exceeded by the user. The soft limit must be less than the hard limit.

Once the user exceeds the soft limit, a timer begins. While the timer is ticking, the user is allowed to operate above the soft limit but cannot exceed the hard limit. Once the user goes below the soft limit, the timer gets reset. However, if the user’s usage remains above the soft limit when the timer expires, the soft limit is enforced as a hard limit. By default, the soft limit timer is seven days.

The value of the timer is shown by the timeleft field in the repquota and quota commands.

For example, let’s say a user has a soft limit of 10,000 blocks and a hard limit of 12,000 blocks. If the user’s block usage exceeds 10,000 blocks and the timer is also exceeded (more than seven days), the user will not be able to allocate more disk blocks on that file system until his or her usage drops below the soft limit.

Core

core(5) — Linux manual page

官方手册里面讲了 core dump 的一些配置和行为特征，简单总结一下。

1. 什么情况下，不会生成 core dump。
2. core dump 的名字配置 /proc/sys/kernel/core_pattern 主要通过这个配置 /proc/sys/kernel/core_uses_pid 一个更基础（更简单？）的配置 但这里有些疑惑，如果 pattern 不是用的默认值，那 used_pid 是不是还生效呢？
3. 将 core 文件管道到另外的程序
4. 哪些 mapping 可以被写到 dump 文件
5. 使用 systemd 做管道以及相应的配置和操作

注1 针对第二点测试了一下，如果 pattern 里面使用了变量，就会忽略 used_pid，如果没有使用变量，就会使用 pattern 中的配置+used_pid

Linux Memory Contoller

Memory Resource Controller - The Linux Kernel documentation

Each cgroup maintains a per cgroup LRU which has the same structure as global VM. When a cgroup goes over its limit, we first try to reclaim memory from the cgroup so as to make space for the new pages that the cgroup has touched. If the reclaim is unsuccessful, an OOM routine is invoked to select and kill the bulkiest task in the cgroup. (See 10. OOM Control below.)

OOM Control¶

memory.oom_control file is for OOM notification and other controls.

Memory cgroup implements OOM notifier using the cgroup notification API (See Control Groups). It allows to register multiple OOM notification delivery and gets notification when OOM happens.

To register a notifier, an application must:

create an eventfd using eventfd(2)

open memory.oom_control file

write string like “ ” to cgroup.event_control

The application will be notified through eventfd when OOM happens. OOM notification doesn’t work for the root cgroup.

You can disable the OOM-killer by writing “1” to memory.oom_control file, as:

#echo 1 > memory.oom_control

If OOM-killer is disabled, tasks under cgroup will hang/sleep in memory cgroup’s OOM-waitqueue when they request accountable memory.

For running them, you have to relax the memory cgroup’s OOM status by

enlarge limit or reduce usage.

To reduce usage,

kill some tasks.

move some tasks to other group with account migration.

remove some files (on tmpfs?)

Then, stopped tasks will work again.

At reading, current status of OOM is shown.

oom_kill_disable 0 or 1 (if 1, oom-killer is disabled)

under_oom 0 or 1 (if 1, the memory cgroup is under OOM, tasks may be stopped.)

oom_kill integer counter The number of processes belonging to this cgroup killed by any kind of OOM killer.

Sb

你怎么傻逼都行，但请不要连累我。

Kafka Healer

https://programmer.group/kafka-golang-client-introduction.html

看到一篇文章，里面有一点点提到了 healer，这介绍还真的是非常准确。

Client name	Advantages and disadvantages
sarama	The number of users is relatively large, but it is relatively difficult to use, with better performance
confluent-kafka-go	The encapsulation of Kafka in C language has strong performance, but depends on librdkafka
kafka-go	The operation is simple, but the performance is poor. The common dual core CPU machine in the production environment only processes about 300 pieces per second
healer	The operation is very simple, and the performance is similar to that of sarama. This product is the work of a big bull of Ctrip. At present, the number of users in the community is small, and there is no corresponding support and maintenance