Jibber jabber toy amazon how to#
Jibber jabber toy amazon software#

Schema is only required if you use a structured db (which is advisable) but it doesn't take that long. Many people will host dbs for you without you having to think about it.

Jibber jabber toy amazon how to#

It's required that you learn how to do this because almost all applications revolve around storing some kind of state and, as was noted, you can't reasonably expect it to persist on the app server without additional ops headaches. Storing data in a database vs in files on disk is like application development 101 and is pretty much a required skill period. If they don't, then it's each time you want it on a new host (which as was correctly pointed out a pain in the ass).

If those hours result in a Dockerfile, then it's the last time. In either case you have to spend hours screwing around with your environment. Once you get it done once, you don't have to screw around with it anymore, and you have it on any box you want in seconds.

At its core Dockerfiles are just shell scripts with worse syntax, so it's not really that much more to learn.

Jibber jabber toy amazon software#

One could say the same about writing the software itself. You 'spend hours messing around' with everything you don't know or understand at first. For training, pay attention to 4-bit bitsandbytes, coming soon: It inferences about 2X slower than exllama from my testing on a RTX 4090, but still about 6X faster than my CPU (Ryzen 5950X).Īgain this is inferencing. Note the latest versions of llama.cpp now have decent GPU support and has both a memory tester and lets you load partial models (n-layers) into your GPU. This repo lists very specific VRAM usage for various LLaMA models (w/ group size, and accounting for context window, which is often missing) - this are all 4-bit GPTQ quantized models: Here's a recent writeup on the LLM performance you can expect for inferencing (training speeds I assume would be similar). Old Nvidia P40 (Pascal 24GB) cards are easily available for $200 or less and would be easy/cheap to play. The more VRAM the better if you'd like to run larger LLMs.