54 Commits

Author	SHA1	Message	Date
perf3ct	a5488771ae	fix showing percentage of embeddings that are completed	2025-03-30 19:43:10 +00:00
perf3ct	40bbdb2faa	fix chunking imports again	2025-03-30 19:41:31 +00:00
perf3ct	6b86bf93ae	fix import paths in chunking	2025-03-30 19:35:07 +00:00
perf3ct	8497e77b55	fix linter errors in providers	2025-03-28 23:27:40 +00:00
perf3ct	2311c3c049	centralize LLM constants more	2025-03-28 23:25:06 +00:00
perf3ct	2899707e64	Better use of interfaces, reducing useage of "any"	2025-03-28 21:47:28 +00:00
perf3ct	44cd2ebda6	fix requeue errors	2025-03-28 20:37:09 +00:00
perf3ct	5456ac32ef	set up embedding similarity constants and similarity system	2025-03-26 23:12:45 +00:00
perf3ct	a7cafceac9	more heavily weigh notes with title matches when giving context to LLM	2025-03-26 23:05:16 +00:00
perf3ct	713805394c	move providers.ts into providers folder	2025-03-26 19:10:16 +00:00
perf3ct	9d29ff4a6c	don't spam the logs if a provider isn't enabled	2025-03-24 21:13:54 +00:00
perf3ct	150b0f0977	remove isEnabled from embedding providers	2025-03-24 20:35:46 +00:00
perf3ct	0707266dc1	reset embedding_queue where objects are "isprocessing"	2025-03-20 22:17:04 +00:00
perf3ct	1a8ce967d9	stop the log spam	2025-03-18 22:20:17 +00:00
perf3ct	f05fe3f72b	set up embedding normalization	2025-03-18 21:09:19 +00:00
perf3ct	08f7f1962b	do a better job with similarity searches	2025-03-18 00:50:55 +00:00
perf3ct	c37201183b	add Voyage AI as Embedding provider	2025-03-17 22:32:00 +00:00
perf3ct	84a8473beb	adapt or regenerate embeddings - allows users to decide	2025-03-17 21:47:11 +00:00
perf3ct	5ad730c153	openai finally works, respect embedding precedence	2025-03-17 21:36:14 +00:00
perf3ct	3268c435e2	improve embedding precedence	2025-03-17 21:03:42 +00:00
perf3ct	d95fd0b049	allow specifying openai embedding models too	2025-03-17 19:54:11 +00:00
perf3ct	d2072c2a6f	"lock" notes that are having their embeddings created	2025-03-16 20:36:47 +00:00
perf3ct	781a2506f0	fix embeddings w/ cls.init()	2025-03-16 18:55:53 +00:00
perf3ct	0081e6f1d0	fix sql error and add missing options	2025-03-16 18:53:01 +00:00
perf3ct	c315b32c99	wait for DB init even to emit before starting LLM services	2025-03-16 18:21:43 +00:00
perf3ct	697d348286	set up more reasonable context window and dimension sizes	2025-03-16 18:08:50 +00:00
perf3ct	8d7e5c8d43	fix the maths for stats	2025-03-12 21:18:59 +00:00
perf3ct	ee7b2283de	correctly style the failed embeddings section	2025-03-12 21:13:49 +00:00
perf3ct	c914aaa4a8	do a better job of handling failed note embeddings	2025-03-12 21:04:06 +00:00
perf3ct	a930b79cf5	synchronize embeddings	2025-03-12 18:22:05 +00:00
perf3ct	b6df3a721c	allow user to select *where* they want to generate embeddings	2025-03-12 18:02:51 +00:00
perf3ct	3fee82eaa5	rename files with the same name	2025-03-12 16:51:09 +00:00
perf3ct	73445d97e7	move chunking to its own folder	2025-03-12 16:33:46 +00:00
perf3ct	46a6533e57	update chunking management	2025-03-12 01:26:25 +00:00
perf3ct	e5afbc6ddc	better manage chunking errors	2025-03-12 01:21:36 +00:00
perf3ct	fcba151287	allow for manual index rebuild, and ONLY rebuild the index	2025-03-12 00:17:30 +00:00
perf3ct	eaa947ef7c	"rebuild index" functionality for users	2025-03-12 00:08:39 +00:00
perf3ct	72b1426d94	break up large vector_store into smaller files	2025-03-12 00:02:02 +00:00
perf3ct	0d2858c7e9	upgrade chunking	2025-03-11 23:04:51 +00:00
perf3ct	1f661e4c90	make sure to not retry chunks if they fail or something else	2025-03-11 20:38:40 +00:00
perf3ct	f47b070f0f	I think this works to handle failed embeddings	2025-03-11 20:22:01 +00:00
perf3ct	71b3b04c53	break up the huge context_extractor into smaller files	2025-03-11 18:39:59 +00:00
perf3ct	0985cec8d6	implement chunking and use becca for some functionality	2025-03-11 18:17:41 +00:00
perf3ct	f482b3b4c8	do a better job of extracting context	2025-03-10 18:53:36 +00:00
perf3ct	adaac46fbf	I'm 100% going to have to destroy this commit later	2025-03-09 02:19:26 +00:00
perf3ct	733fdcf8ba	update relationship weights	2025-03-08 23:36:04 +00:00
perf3ct	7e232d17e1	Create better relationships between notes, sanitize ridiculous spacing to save tokens	2025-03-08 23:34:09 +00:00
perf3ct	51c83bbbc8	show fancier stats	2025-03-08 23:21:21 +00:00
perf3ct	0cd1be5568	Show embedding generation stats to user	2025-03-08 23:17:13 +00:00
perf3ct	6ace4d5692	nearly able to process embeddings	2025-03-08 23:08:25 +00:00