-
Podman Compose Missing Networks
Q: macos上,podman-compose启动失败,报错 “RuntimeError: missing networks: default”
A: 根据搜索结果,Podman Compose 在某些情况下无法自动创建默认网络,需要在 docker-compose.yml 文件中显式定义默认网络。例如:
version: '3.8' services: your_service: image: your_image networks: - default networks: default: driver: bridge
-
Vector Search
https://medium.com/@vidiptvashist/building-a-vector-database-from-scratch-in-python-6bd683ba5171
关键还是如何做 embedding。以及如何做相似度计算?
后面看一下有哪些常用的embedding model 和算法。
还有,代码中都是简单的字符串,如果特别长,怎么切片呢?这个参数是不是也重要?
from typing import Any import numpy as np class VectorStore: def __init__(self): self.vector_data: dict[str, np.ndarray] = {} # A dictionary to store vectors self.vector_index: dict[str, dict] = {} # An indexing structure for retrieval def add_vector(self, vector_id: str, vector: np.ndarray): """ Add a vector to the store. Args: vector_id (str or int): A unique identifier for the vector. vector (numpy.ndarray): The vector data to be stored. """ self.vector_data[vector_id] = vector self._update_index(vector_id, vector) def get_vector(self, vector_id): """ Retrieve a vector from the store. Args: vector_id (str or int): The identifier of the vector to retrieve. Returns: numpy.ndarray: The vector data if found, or None if not found. """ return self.vector_data.get(vector_id) def _update_index(self, vector_id, vector): """ Update the index with the new vector. Args: vector_id (str or int): The identifier of the vector. vector (numpy.ndarray): The vector data. """ # In this simple example, we use brute-force cosine similarity for indexing for existing_id, existing_vector in self.vector_data.items(): similarity = np.dot(vector, existing_vector) / ( np.linalg.norm(vector) * np.linalg.norm(existing_vector) ) if existing_id not in self.vector_index: self.vector_index[existing_id] = {} self.vector_index[existing_id][vector_id] = similarity def find_similar_vectors(self, query_vector, num_results=5): """ Find similar vectors to the query vector using brute-force search. Args: query_vector (numpy.ndarray): The query vector for similarity search. num_results (int): The number of similar vectors to return. Returns: list: A list of (vector_id, similarity_score) tuples for the most similar vectors. """ results: list[tuple[str, float]] = [] for vector_id, vector in self.vector_data.items(): similarity = np.dot(query_vector, vector) / ( np.linalg.norm(query_vector) * np.linalg.norm(vector) ) results.append((vector_id, similarity)) # Sort by similarity in descending order results.sort(key=lambda x: x[1], reverse=True) # Return the top N results return results[:num_results] # Establish a VectorStore instance vector_store = VectorStore() # Creating an instance of the VectorStore class # Define sentences sentences = [ # Defining a list of example sentences "I eat mango", "mango is my favorite fruit", "mango, apple, oranges are fruits", "fruits are good for health", ] # Tokenization and Vocabulary Creation vocabulary: set[str] = set() # Initializing an empty set to store unique words for sentence in sentences: # Iterating over each sentence in the list tokens = ( sentence.lower().split() ) # Tokenizing the sentence by splitting on whitespace and converting to lowercase vocabulary.update(tokens) # Updating the set of vocabulary with unique tokens # Assign unique indices to vocabulary words word_to_index = { word: i for i, word in enumerate(vocabulary) } # Creating a dictionary mapping words to unique indices # Vectorization # Initializing an empty dictionary to store sentence vectors sentence_vectors: dict[str, np.ndarray] = {} for sentence in sentences: # Iterating over each sentence in the list tokens = ( sentence.lower().split() ) # Tokenizing the sentence by splitting on whitespace and converting to lowercase vector = np.zeros( len(vocabulary) ) # Initializing a numpy array of zeros for the sentence vector for token in tokens: # Iterating over each token in the sentence vector[ word_to_index[token] ] += 1 # Incrementing the count of the token in the vector sentence_vectors[sentence] = ( vector # Storing the vector for the sentence in the dictionary ) # Store in VectorStore for sentence, vector in sentence_vectors.items(): # Iterating over each sentence vector vector_store.add_vector( sentence, vector ) # Adding the sentence vector to the VectorStore # Similarity Search query_sentence = "Mango is the best fruit" # Defining a query sentence query_vector = np.zeros( len(vocabulary) ) # Initializing a numpy array of zeros for the query vector query_tokens = ( query_sentence.lower().split() ) # Tokenizing the query sentence and converting to lowercase for token in query_tokens: # Iterating over each token in the query sentence if token in word_to_index: # Checking if the token is present in the vocabulary query_vector[ word_to_index[token] ] += 1 # Incrementing the count of the token in the query vector similar_sentences = vector_store.find_similar_vectors( query_vector, num_results=2 ) # Finding similar sentences # Display similar sentences print("Query Sentence:", query_sentence) # Printing the query sentence print("Similar Sentences:") # Printing the header for similar sentences for ( sentence, similarity, ) in similar_sentences: # Iterating over each similar sentence and its similarity score print( f"{sentence}: Similarity = {similarity:.4f}" ) # Printing the similar sentence and its similarity score
-
这不是有效沟通
A: Deepseek 为什么成本这么低? B:我不懂。 A:你不懂?你们搞技术的应该都懂啊。 B:我不懂。 A:那你懂啥?
-
Shortcut Connection In Deep Nn
In conclusion, shortcut connections are important for overcoming the limitations posed by the vanishing gradient problem in deep neural networks. Shortcut connections are a core building block of very large models such as LLMs, and they will help facilitate more effective training by ensuring consistent gradient flow across layers when we train the GPT model in the next chapter.
总之,捷径连接对于克服深度神经网络中梯度消失问题带来的限制非常重要。快捷连接是超大型模型的核心构建块,例如LLMs,当我们在下一章训练 GPT 模型时,它们将通过确保跨层的梯度流一致来帮助促进更有效的训练。
原理是什么?
为什么梯度会消失,为什么捷径连接可以解决这个问题?
-
认错
我觉得人生过的顺畅一些的一个改变是,把第一反应从辩解(缺少正确逻辑的辩解)变成认错。
-
Note2 Build A Llm From Scratch
需要看pytorch的文档,进一步了解里面第一步的作用是什么,以了解MultiHeadAttention的原理
class MultiHeadAttention(nn.Module): def __init__(self, d_in, d_out, context_length, dropout, num_heads, qkv_bias=False): super().__init__() assert d_out % num_heads == 0, "d_out must be divisible by num_heads" self.d_out = d_out self.num_heads = num_heads self.head_dim = d_out // num_heads # 1 self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias) self.W_key = nn.Linear(d_in, d_out, bias=qkv_bias) self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias) self.out_proj = nn.Linear(d_out, d_out) # 2 self.dropout = nn.Dropout(dropout) self.register_buffer( "mask", torch.triu(torch.ones(context_length, context_length), diagonal=1) ) def forward(self, x): b, num_tokens, d_in = x.shape keys = self.W_key(x) # 3 queries = self.W_query(x) # 3 values = self.W_value(x) # 3 keys = keys.view(b, num_tokens, self.num_heads, self.head_dim) # 4 values = values.view(b, num_tokens, self.num_heads, self.head_dim) queries = queries.view(b, num_tokens, self.num_heads, self.head_dim) keys = keys.transpose(1, 2) # 5 queries = queries.transpose(1, 2) # 5 values = values.transpose(1, 2) # 5 attn_scores = queries @ keys.transpose(2, 3) # 6 mask_bool = self.mask.bool()[:num_tokens, :num_tokens] # 7 attn_scores.masked_fill_(mask_bool, -torch.inf) # 8 attn_weights = torch.softmax(attn_scores / keys.shape[-1] ** 0.5, dim=-1) attn_weights = self.dropout(attn_weights) context_vec = (attn_weights @ values).transpose(1, 2) # 9 # 10 context_vec = context_vec.contiguous().view(b, num_tokens, self.d_out) context_vec = self.out_proj(context_vec) # 11 return context_vec
-
Note1 Build A Llm From Scratch
-
一句话,像
"I am a student."
,am 的 tokenID 是一个[xx,xx,xx]这样的“三维”向量。第书的第二章讲的encode token,是变成一个数字,怎么到了第三章,突然成了一个三维向量?中间发生了哪些操作? -
在第三章的“Implementing self-attention with trainable weights”这一节,说到“我们成功将6个输入标记从三维投影到二维嵌入空间上”。为什么要把三维转成二维?这里说的“embedding space”又是指什么?
-
-
Git Pull Pr
git fetch origin pull/265/head:pr-265
-
Github Pages Cloudflare
将 GitHub Pages 的自定义域名迁到 cloudflare 之后,一直死循环 301。
原因是,cloudflare 默认的 SSL 配置是 “灵活”:仅在访问者与 Cloudflare 之间启用加密。这可以避免浏览器发出安全警告,但 Cloudflare 与您的源服务器之间的所有连接均通过 HTTP 建立。
而我的 github pages 配置了 enforce https,所以就会出现死循环。
但是,还有一个github pages 项目使用不能配置 enforce https,配置页面显示”Enforce HTTPS — Unavailable for your site because your domain is not properly configured to support HTTPS”,还不明白怎么回事,怎么解决。目前是在 cloudflare 配置了始终重定向到 https
-
Autogen Question 1
autogen里面的一段代码如下:
selector_group_chat = SelectorGroupChat( [add_agent, multiply_agent, subtract_agent, divide_agent, identity_agent], model_client=OpenAIChatCompletionClient(model="gpt-4o"), termination_condition=termination_condition, allow_repeated_speaker=True, # Allow the same agent to speak multiple times, necessary for this task. selector_prompt=( "Available roles:\n{roles}\nTheir job descriptions:\n{participants}\n" "Current conversation history:\n{history}\n" "Please select the most appropriate role for the next message, and only return the role name." ), )
selector_prompt 里面的 participants 是什么意思?