New Release 2023.07.30 (IBM#108)

Merging develop into main.

Author: onkarbhardwaj

EXTENSIONS.md

@@ -23,7 +23,7 @@ IBM Generative AI extensions can be either of the following :
 Extensions that are meant for public use from the get-go should instead be developed as official open-source extensions. Examples of official extensions that have already been released are LangChain, Pandas, and Hugging Face extensions.
 
 ### Ownership and location
-Open-source extensions should be submitted directly to the [IBM Generative AI open-source repository](https://github.com/IBM/ibm-generative-ai) and should be developed following the [open-source Gen AI contribution guide](https://github.com/IBM/ibm-generative-ai/blob/main/DEVELOPMENT.md). Open-source official extensions are typically developed by the Gen AI team, or in collaboration with them. Providing maintenance to open-source official extensions is responsability of the Gen AI team.
+Open-source extensions should be submitted directly to the [IBM Generative AI open-source repository](https://github.com/IBM/ibm-generative-ai) and should be developed following the [open-source Gen AI contribution guide](https://github.com/IBM/ibm-generative-ai/blob/main/CONTRIBUTING.md). Open-source official extensions are typically developed by the Gen AI team, or in collaboration with them. Providing maintenance to open-source official extensions is responsability of the Gen AI team.
 
 ## Open-source "third-party" extensions
 All other extensions neither implemented nor officially maintained by the Gen AI team are referred to as open-source third-party extensions.

GETTING_STARTED.md

@@ -2,7 +2,6 @@
 
 ## <a name='TableofContents'></a>Table of Contents
 
-<!-- vscode-markdown-toc -->
 * [Table of Contents](#table-of-contents)
 * [Installation](#installation)
 * [Gen AI Endpoint](#gen-ai-endpoint)
@@ -11,6 +10,7 @@
     * [Async Example](#async-example)
     * [Synchronous Example](#synchronous-example)
 * [Tips and Troubleshooting](#tips-and-troubleshooting)
+    * [Model Availability](#model-availability)
     * [Enabling Logs](#enabling-logs)
     * [Experimenting with a Large Number of Prompts](#many-prompts)
 * [Extensions](#extensions)
@@ -22,10 +22,13 @@
 ```bash
 pip install ibm-generative-ai
 ```
+
 #### <a name='KnownIssueFixes:'></a>Known Issue Fixes:
+
 - **[SSL Issue]** If you run into "SSL_CERTIFICATE_VERIFY_FAILED" please run the following code snippet here: [support](SUPPORT.md).
 
 ### <a name='Prerequisites'></a>Prerequisites
+
 Python version >= 3.9
 
 Pip version >= 22.0.1
@@ -70,13 +73,11 @@ creds = Credentials(api_key=my_api_key, api_endpoint=my_api_endpoint)
 
 ```
 
-
 ## <a name='Examples'></a>Examples
 
 There are a number of examples you can try in the [`examples/user`](examples/user) directory.
 Login to [workbench.res.ibm.com](https://workbench.res.ibm.com/) and get your GenAI API key. Then, create a `.env` file and assign the `GENAI_KEY` value as below example. [More information](#gen-ai-endpoint)
 
-
 ```ini
 GENAI_KEY=YOUR_GENAI_API_KEY
 # GENAI_API=GENAI_API_ENDPOINT << for a different endpoint
@@ -184,6 +185,40 @@ for response in responses:
 
 ## <a name='TipsAndTroubleshooting'></a>Tips and Troubleshooting
 
+### <a name='Model Availability'></a>Model Availability
+To test the reachability of your endpoint and availability of desired model, use the following utility script with your model details:
+```python
+import os
+
+from dotenv import load_dotenv
+
+from genai.credentials import Credentials
+from genai.model import Model
+
+# make sure you have a .env file under genai root with
+# GENAI_KEY=<your-genai-key>
+# GENAI_API=<your-genai-api endpoint>
+load_dotenv()
+api_key = os.getenv("GENAI_KEY", None)
+api_url = os.getenv("GENAI_API", None)
+creds = Credentials(api_key, api_endpoint=api_url)
+
+print("======= List of all available models =======")
+for m in Model.models(credentials=creds):
+    print(m)
+
+print("====== Checking availability of a specific model =======")
+model_id = "<string-id-of-model>"
+model = Model(model_id, params=None, credentials=creds)
+print(f"Model availability for {model_id}: {model.available()}")
+
+print("====== Display model card =======")
+model = Model(model_id, params=None, credentials=creds)
+model_info = model.info()
+print(f"Model info for {model_id}: \n{model_info}")
+print(f"Extract fields from model card (e.g., token_limit): {model_info.token_limit}")
+```
+
 ### <a name='EnablingLogs'></a>Enabling Logs
 
 If you're building an application or example and would like to see the GENAI logs, you can enable them in the following way:
@@ -223,6 +258,7 @@ To learn more about logging in python, you can follow the tutorial [here](https:
 
 Since generating responses for a large number of prompts can be time-consuming and there could be unforeseen circumstances such as internet connectivity issues, here are some strategies
 to work with:
+
 - Start with a small number of prompts to prototype the code. You can enable logging as described above for debugging during prototyping.
 - Include exception handling in sensitive sections such as callbacks.
 - Checkpoint/save prompts and received responses periodically.
@@ -257,10 +293,13 @@ us if you want support for some framework as an extension or want to design an e
 ### <a name='LangChainExtension'></a>LangChain Extension
 
 Install the langchain extension as follows:
+
 ```bash
 pip install "ibm-generative-ai[langchain]"
 ```
+
 Currently the langchain extension allows IBM Generative AI models to be wrapped as Langchain LLMs and translation between genai PromptPatterns and LangChain PromptTemplates. Below are sample snippets
+
 ```python
 import os
 from dotenv import load_dotenv
@@ -292,13 +331,6 @@ print(langchain_model(template.format(question="What is life?")))
 print(genai_model.generate([pattern.sub("question", "What is life?")])[0].generated_text)
 ```
 
-## <a name='[Deprecated] Model Types'></a>[Deprecated] Model Types
-
-Model types can be imported from the [ModelType class](src/genai/schemas/models.py). If you want to use a model that is not included in this class, you can pass it as a string as exemplified [here](src/genai/schemas/models.py).
-
-Models can be selected by passing their string id to the Model class as exemplified [here](src/genai/schemas/models.py).
-
-
 ## <a name='Support'></a>Support
 
 Need help? Check out how to get [support](SUPPORT.md)

documentation/docs/source/rst_source/genai.schemas.models.rst

@@ -10,7 +10,6 @@ Submodules
    genai.schemas.descriptions
    genai.schemas.generate_params
    genai.schemas.history_params
-   genai.schemas.models
    genai.schemas.responses
    genai.schemas.token_params
    genai.schemas.tunes_params

documentation/docs/source/rst_source/genai.schemas.rst

@@ -7,7 +7,7 @@
 from dotenv import load_dotenv
 
 from genai.model import Credentials, Model
-from genai.schemas import GenerateParams, ModelType, TokenParams
+from genai.schemas import GenerateParams, TokenParams
 from genai.services.connection_manager import ConnectionManager
 from genai.services.request_handler import RequestHandler
 
@@ -80,7 +80,7 @@ async def flaky_async_generate(
 tokenize_params = TokenParams(return_tokens=True)
 
 
-flan_ul2 = Model(ModelType.FLAN_UL2, params=generate_params, credentials=creds)
+flan_ul2 = Model("google/flan-ul2", params=generate_params, credentials=creds)
 prompts = ["Generate a random number > {}: ".format(i) for i in range(25)]
 for response in flan_ul2.generate_async(prompts, ordered=True):
     pass

examples/dev/async-flaky-request-handler.py

@@ -6,7 +6,7 @@
 from dotenv import load_dotenv
 
 from genai.model import Credentials, GenAiException, Model
-from genai.schemas import GenerateParams, ModelType, TokenParams
+from genai.schemas import GenerateParams, TokenParams
 from genai.services.async_generator import AsyncResponseGenerator
 
 num_requests = 0
@@ -83,7 +83,7 @@ def tokenize_async(self, prompts, ordered=False, callback=None, options=None):
 tokenize_params = TokenParams(return_tokens=True)
 
 
-flan_ul2 = FlakyModel(ModelType.FLAN_UL2_20B, params=generate_params, credentials=creds)
+flan_ul2 = FlakyModel("google/flan-ul2", params=generate_params, credentials=creds)
 prompts = ["Generate a random number > {}: ".format(i) for i in range(17)]
 print("======== Async Generate with ordered=True ======== ")
 counter = 0
@@ -97,7 +97,7 @@ def tokenize_async(self, prompts, ordered=False, callback=None, options=None):
 num_requests = 0
 
 # Instantiate a model proxy object to send your requests
-flan_ul2 = FlakyModel(ModelType.FLAN_UL2_20B, params=tokenize_params, credentials=creds)
+flan_ul2 = FlakyModel("google/flan-ul2", params=tokenize_params, credentials=creds)
 prompts = ["Generate a random number > {}: ".format(i) for i in range(23)]
 print("======== Async Tokenize with ordered=True ======== ")
 counter = 0

examples/dev/async-flaky-responses-ordered.py

@@ -3,7 +3,7 @@
 from dotenv import load_dotenv
 
 from genai.model import Credentials, Model
-from genai.schemas import GenerateParams, ModelType
+from genai.schemas import GenerateParams
 
 # make sure you have a .env file under genai root with
 # GENAI_KEY=<your-genai-key>
@@ -24,7 +24,8 @@
     " during iteration it will do symb1 symb1 symb1 due to how it"
     " maps internally. ===="
 )
-for key, modelid in ModelType.__members__.items():
+for model_card in Model.models(credentials=creds):
+    modelid = model_card.id
     model = Model(modelid, params=params, credentials=creds)
     responses = [response.generated_text for response in model.generate(prompts)]
     print(modelid, ":", responses)

examples/dev/generate-all-models.py

@@ -4,7 +4,7 @@
 from dotenv import load_dotenv
 
 from genai.model import Credentials, Model
-from genai.schemas import GenerateParams, ModelType
+from genai.schemas import GenerateParams
 
 logging.basicConfig(level=logging.INFO)
 
@@ -22,7 +22,7 @@
 params = GenerateParams(decoding_method="sample", max_new_tokens=10)
 
 # Instantiate a model proxy object to send your requests
-flan_ul2 = Model(ModelType.FLAN_UL2, params=params, credentials=creds)
+flan_ul2 = Model("google/flan-ul2", params=params, credentials=creds)
 
 prompts = ["Hello! How are you?", "How's the weather?"]
 for response in flan_ul2.generate_async(prompts):

examples/dev/logging_example.py

@@ -1,3 +1,3 @@
 torch==2.0.1
-transformers==4.29.2
+transformers==4.30.0
 sentencepiece==0.1.99

examples/user/localserver/requirements.txt

@@ -0,0 +1,29 @@
+import os
+
+from dotenv import load_dotenv
+
+from genai.credentials import Credentials
+from genai.model import Model
+
+# make sure you have a .env file under genai root with
+# GENAI_KEY=<your-genai-key>
+# GENAI_API=<your-genai-api endpoint>
+load_dotenv()
+api_key = os.getenv("GENAI_KEY", None)
+api_url = os.getenv("GENAI_API", None)
+creds = Credentials(api_key, api_endpoint=api_url)
+
+print("======= List of all models =======")
+for m in Model.models(credentials=creds):
+    print(m)
+
+print("====== Checking model availability =======")
+model = Model("google/ul2", params=None, credentials=creds)
+print("Model availability for 'google/ul2': ", model.available())
+
+model = Model("random", params=None, credentials=creds)
+print("Model availability for 'random': ", model.available())
+
+print("====== Display model card =======")
+model = Model("google/ul2", params=None, credentials=creds)
+print("Model info for 'google/ul2': \n", model.info())

examples/user/model_utils.py

@@ -2,9 +2,10 @@
 
 from dotenv import load_dotenv
 
-from genai.model import Credentials, Model
+from genai.credentials import Credentials
+from genai.model import Model
 from genai.prompt_pattern import PromptPattern
-from genai.schemas import GenerateParams, ModelType
+from genai.schemas import GenerateParams
 
 # make sure you have a .env file under genai root with
 # GENAI_KEY=<your-genai-key>
@@ -15,7 +16,7 @@
 creds = Credentials(api_key, api_endpoint=api_url)
 params = GenerateParams(temperature=0.5)
 
-model = Model(ModelType.FLAN_UL2, params=params, credentials=creds)
+model = Model("google/flan-ul2", params=params, credentials=creds)
 
 
 _template = """

examples/user/prompt_templating/watsonx-prompt-output.py

@@ -3,10 +3,11 @@
 
 from dotenv import load_dotenv
 
-from genai.model import Credentials, Model
+from genai.credentials import Credentials
+from genai.model import Model
 from genai.options import Options
 from genai.prompt_pattern import PromptPattern
-from genai.schemas import GenerateParams, ModelType
+from genai.schemas import GenerateParams
 
 # make sure you have a .env file under genai root with
 # GENAI_KEY=<your-genai-key>
@@ -18,7 +19,7 @@
 creds = Credentials(api_key, api_endpoint=api_url)
 params = GenerateParams(temperature=0.5)
 
-model = Model(ModelType.FLAN_UL2, params=params, credentials=creds)
+model = Model("google/flan-ul2", params=params, credentials=creds)
 
 
 _template = """

examples/user/prompt_templating/watsonx-prompt-pattern-ux-async.py

@@ -3,10 +3,11 @@
 
 from dotenv import load_dotenv
 
-from genai.model import Credentials, Model
+from genai.credentials import Credentials
+from genai.model import Model
 from genai.options import Options
 from genai.prompt_pattern import PromptPattern
-from genai.schemas import GenerateParams, ModelType
+from genai.schemas import GenerateParams
 
 # make sure you have a .env file under genai root with
 # GENAI_KEY=<your-genai-key>
@@ -18,7 +19,7 @@
 creds = Credentials(api_key, api_endpoint=api_url)
 params = GenerateParams(temperature=0.5)
 
-model = Model(ModelType.FLAN_UL2, params=params, credentials=creds)
+model = Model("google/flan-ul2", params=params, credentials=creds)
 
 
 _template = """

examples/user/prompt_templating/watsonx-prompt-pattern-ux.py

@@ -110,28 +110,40 @@ def get_creds():
         if status in ["FAILED", "HALTED"]:
             print("Model tuning failed or halted")
         else:
+            print("Model info:\n")
+            print(tuned_model.info())
+            time.sleep(5)
+
             prompt = input("Enter a prompt:\n")
             genparams = GenerateParams(
                 decoding_method="greedy",
                 max_new_tokens=50,
                 min_new_tokens=1,
             )
             print("Answer = ", tuned_model.generate([prompt])[0].generated_text)
+            time.sleep(5)
+
+            print("~~~~~~~ List of all models ~~~~~~")
+            for m in Model.models(credentials=creds):
+                print(m, "\n")
+            time.sleep(10)
 
-            print("~~~~~~~ Listing tunes and getting tune metadata with TuneManager ~~~~~")
+            print("~~~~~~~ Getting list of all tuned models with TuneManager ~~~~~")
 
-            list_params = TunesListParams(limit=5, offset=0)
+            list_params = TunesListParams(limit=50, offset=0)
 
             tune_list = TuneManager.list_tunes(credentials=creds, params=list_params)
             print("\n\nList of tunes: \n\n")
             for tune in tune_list.results:
                 print(tune, "\n")
+            time.sleep(10)
 
             tune_get_result = TuneManager.get_tune(credentials=creds, tune_id=tuned_model.model)
             print(
                 "\n\n~~~~~ Metadata for a single tune with TuneManager ~~~~: \n\n",
                 tune_get_result,
             )
+            time.sleep(5)
 
             print("~~~~~~~ Deleting a tuned model ~~~~~")
             to_delete = input("Delete this model? (y/N):\n")