Skip to content

Commit 5ca325c

Browse files
committed
docs: refactoring of the doc
1 parent f968065 commit 5ca325c

File tree

3 files changed

+71
-33
lines changed

3 files changed

+71
-33
lines changed

docs/source/introduction/overview.rst

+70-9
Original file line numberDiff line numberDiff line change
@@ -30,37 +30,93 @@ ScrapGraphAI supports a wide range of AI models from various providers. Each mod
3030
OpenAI Models
3131
-------------
3232
- GPT-3.5 Turbo (16,385 tokens)
33-
- GPT-4 (8,192 tokens)
33+
- GPT-3.5 (4,096 tokens)
34+
- GPT-3.5 Turbo Instruct (4,096 tokens)
3435
- GPT-4 Turbo Preview (128,000 tokens)
35-
- GPT-4o (128000 tokens)
36-
- GTP-4o-mini (128000 tokens)
36+
- GPT-4 Vision Preview (128,000 tokens)
37+
- GPT-4 (8,192 tokens)
38+
- GPT-4 32k (32,768 tokens)
39+
- GPT-4o (128,000 tokens)
40+
- O1 Preview (128,000 tokens)
41+
- O1 Mini (128,000 tokens)
3742

3843
Azure OpenAI Models
3944
-------------------
4045
- GPT-3.5 Turbo (16,385 tokens)
41-
- GPT-4 (8,192 tokens)
46+
- GPT-3.5 (4,096 tokens)
4247
- GPT-4 Turbo Preview (128,000 tokens)
43-
- GPT-4o (128000 tokens)
44-
- GTP-4o-mini (128000 tokens)
48+
- GPT-4 (8,192 tokens)
49+
- GPT-4 32k (32,768 tokens)
50+
- GPT-4o (128,000 tokens)
51+
- O1 Preview (128,000 tokens)
52+
- O1 Mini (128,000 tokens)
4553

4654
Google AI Models
4755
----------------
4856
- Gemini Pro (128,000 tokens)
57+
- Gemini 1.5 Flash (128,000 tokens)
4958
- Gemini 1.5 Pro (128,000 tokens)
59+
- Gemini 1.0 Pro (128,000 tokens)
5060

5161
Anthropic Models
5262
----------------
5363
- Claude Instant (100,000 tokens)
54-
- Claude 2 (200,000 tokens)
64+
- Claude 2 (9,000 tokens)
65+
- Claude 2.1 (200,000 tokens)
5566
- Claude 3 (200,000 tokens)
67+
- Claude 3.5 (200,000 tokens)
68+
- Claude 3 Opus (200,000 tokens)
69+
- Claude 3 Sonnet (200,000 tokens)
70+
- Claude 3 Haiku (200,000 tokens)
5671

5772
Mistral AI Models
5873
-----------------
59-
- Mistral Large (128,000 tokens)
74+
- Mistral Large Latest (128,000 tokens)
75+
- Open Mistral Nemo (128,000 tokens)
76+
- Codestral Latest (32,000 tokens)
6077
- Open Mistral 7B (32,000 tokens)
6178
- Open Mixtral 8x7B (32,000 tokens)
79+
- Open Mixtral 8x22B (64,000 tokens)
80+
- Open Codestral Mamba (256,000 tokens)
6281

63-
For a complete list of supported models and their token limits, please refer to the API documentation.
82+
Ollama Models
83+
-------------
84+
- Command-R (12,800 tokens)
85+
- CodeLlama (16,000 tokens)
86+
- DBRX (32,768 tokens)
87+
- DeepSeek Coder 33B (16,000 tokens)
88+
- Llama2 Series (4,096 tokens)
89+
- Llama3 Series (8,192-128,000 tokens)
90+
- Mistral Models (32,000-128,000 tokens)
91+
- Mixtral 8x22B Instruct (65,536 tokens)
92+
- Phi3 Series (12,800-128,000 tokens)
93+
- Qwen Series (32,000 tokens)
94+
95+
Hugging Face Models
96+
------------------
97+
- Grok-1 (8,192 tokens)
98+
- Meta Llama 3 Series (8,192 tokens)
99+
- Google Gemma Series (8,192 tokens)
100+
- Microsoft Phi Series (2,048-131,072 tokens)
101+
- GPT-2 Series (1,024 tokens)
102+
- DeepSeek V2 Series (131,072 tokens)
103+
104+
Bedrock Models
105+
-------------
106+
- Claude 3 Series (200,000 tokens)
107+
- Llama2 & Llama3 Series (4,096-8,192 tokens)
108+
- Mistral Series (32,768 tokens)
109+
- Titan Embed Text (8,000 tokens)
110+
- Cohere Embed (512 tokens)
111+
112+
Fireworks Models
113+
---------------
114+
- Llama V2 7B (4,096 tokens)
115+
- Mixtral 8x7B Instruct (4,096 tokens)
116+
- Llama 3.1 Series (131,072 tokens)
117+
- Mixtral MoE Series (65,536 tokens)
118+
119+
For a complete and up-to-date list of supported models and their token limits, please refer to the API documentation.
64120

65121
Understanding token limits is crucial for optimizing your scraping tasks. Larger token limits allow for processing more text in a single API call, which can be beneficial for scraping lengthy web pages or documents.
66122

@@ -139,3 +195,8 @@ Sponsors
139195
:width: 15%
140196
:alt: Stat Proxies
141197
:target: https://dashboard.statproxies.com/?refferal=scrapegraph
198+
199+
.. image:: ../../assets/scrapedo.png
200+
:width: 11%
201+
:alt: Scrapedo
202+
:target: https://scrape.do

docs/source/modules/scrapegraphai.helpers.models_tokens.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ Example usage:
1919
print(f"GPT-4 token limit: {gpt4_limit}")
2020
2121
# Check the token limit for a specific model
22-
model_name = "gpt-3.5-turbo"
22+
model_name = "gpt-4o-mini"
2323
if model_name in models_tokens['openai']:
2424
print(f"{model_name} token limit: {models_tokens['openai'][model_name]}")
2525
else:

docs/source/scrapers/benchmarks.rst

-23
This file was deleted.

0 commit comments

Comments
 (0)