diff --git a/ml/cloud_shell_tutorials/cloud-nl-intro/tutorial.md b/ml/cloud_shell_tutorials/cloud-nl-intro/tutorial.md index 01fb4e2..d144f23 100644 --- a/ml/cloud_shell_tutorials/cloud-nl-intro/tutorial.md +++ b/ml/cloud_shell_tutorials/cloud-nl-intro/tutorial.md @@ -29,7 +29,8 @@ Return to this tab once you're done. ## Set your project in the Cloud Shell and create an API Key -First, run the following command to ensure that the Cloud Shell is using the correct GCP project— whether new or existing— as follows (replacing `` with the name of your project): +First, run the following command to ensure that the Cloud Shell is using the correct GCP project +(replacing `` with the name of your project): ```bash gcloud config set project @@ -37,7 +38,7 @@ First, run the following command to ensure that the Cloud Shell is using the cor Next, since we'll be using curl to send a request to the Natural Language API, we'll need to generate an API key to pass in our request URL. -> **Note**: If you've already created an API key in this project during one of the other Cloud Shell tutorials, you can just use the existing key⸺you don't need to create another one. +> **Note**: If you've already created an API key in this project during one of the other Cloud Shell tutorials, you can just use the existing key⸺you don't need to create another one. Just be sure to set the `API_KEY` environment variable with your existing key as described below. To create an API key, navigate to: @@ -65,7 +66,9 @@ Next, you'll enable the Natural Language API for your project, if you've not alr ## Enable the Natural Langage API -Click on [this link](https://console.cloud.google.com/flows/enableapi?apiid=language.googleapis.com) to enable the Natural Language API for your project. (After you've enabled it, you don't need to do any further setup, as you've already set up an API key above.) +Click on [this link](https://console.cloud.google.com/flows/enableapi?apiid=language.googleapis.com) to enable the Natural Language API for your project. + +After you've enabled it, you don't need to do any further setup, as you've already set up an API key. Just return to this tab. Next, you'll use the Natural Language API to analyze *entities* in text. diff --git a/ml/cloud_shell_tutorials/cloud-nl-text-classification/tutorial.md b/ml/cloud_shell_tutorials/cloud-nl-text-classification/tutorial.md index 92ed855..1b02ede 100644 --- a/ml/cloud_shell_tutorials/cloud-nl-text-classification/tutorial.md +++ b/ml/cloud_shell_tutorials/cloud-nl-text-classification/tutorial.md @@ -27,7 +27,8 @@ Return to this tab once you're done. ## Set your project in the Cloud Shell and create an API Key -First, run the following command to ensure that the Cloud Shell is using the correct GCP project— whether new or existing— as follows (replacing `` with the name of your project): +First, run the following command to ensure that the Cloud Shell is using the correct GCP project +(replacing `` with the name of your project): ```bash gcloud config set project @@ -35,7 +36,7 @@ First, run the following command to ensure that the Cloud Shell is using the cor Next, since we'll be using curl to send a request to the Natural Language API, we'll need to generate an API key to pass in our request URL. -> **Note**: If you've already created an API key in this project during one of the other Cloud Shell tutorials, you can just use the existing key⸺you don't need to create another one. +> **Note**: If you've already created an API key in this project during one of the other Cloud Shell tutorials, you can just use the existing key⸺you don't need to create another one. Just be sure to set the `API_KEY` environment variable with your existing key as described below. To create an API key, navigate to: @@ -63,7 +64,9 @@ Next, you'll enable the Natural Language API for your project, if you've not alr ## Enable the Natural Langage API -Click on [this link](https://console.cloud.google.com/flows/enableapi?apiid=language.googleapis.com) to enable the Natural Language API for your project. (After you've enabled it, you don't need to do any further setup, as you've already set up an API key above.) +Click on [this link](https://console.cloud.google.com/flows/enableapi?apiid=language.googleapis.com) to enable the Natural Language API for your project. + +After you've enabled it, you don't need to do any further setup, as you've already set up an API key. Just return to this tab. Next, you'll use the Natural Language API's `classifyText` method to classify a news article. @@ -81,7 +84,7 @@ Using the Natural Language API's `classifyText` method, we can sort our text dat We'll start by classifying a single article, and then we'll see how we can use this method to make sense of a large news corpus. To start, let's take this headline and description from a New York Times article in the food section: -> *A Smoky Lobster Salad With a Tapa Twist. This spin on the Spanish pulpo a la gallega skips the octopus, but keeps the sea salt, olive oil, pimentón and boiled potatoes.* +> *A Smoky Lobster Salad With a Tapa Twist. This spin on the Spanish pulpo a la gallega skips the octopus, but keeps the sea salt, olive oil, pimentón and boiled potatoes.* Bring up the `request.json` file `walkthrough editor-open-file "code-snippets/ml/cloud_shell_tutorials/cloud-nl-text-classification/request.json" "in the text editor"`. @@ -110,17 +113,17 @@ Notice that the curl command used the API key that you generated. Let's take a look at the response: ```json -{ categories: - [ - { +{ categories: + [ + { name: '/Food & Drink/Cooking & Recipes', - confidence: 0.85 + confidence: 0.85 }, - { + { name: '/Food & Drink/Food/Meat & Seafood', - confidence: 0.63 + confidence: 0.63 } - ] + ] } ``` @@ -131,9 +134,9 @@ Classifying a single article is cool, but to really see the power of this featur ## Classifying a large text dataset -To see how the `classifyText` method can help us understand a dataset with lots of text, we'll use this [public dataset](http://mlg.ucd.ie/datasets/bbc.html) of BBC news articles. The dataset consists of 2,225 articles in five topic areas (business, entertainment, politics, sport, tech) from 2004 - 2005. We've put a subset of these articles into a public [Google Cloud Storage](https://cloud.google.com/storage/) (GCS) bucket. Each of the articles is in a `.txt` file. +To see how the `classifyText` method can help us understand a dataset with lots of text, we'll use this [public dataset](http://mlg.ucd.ie/datasets/bbc.html) of BBC news articles. The dataset consists of 2,225 articles in five topic areas (business, entertainment, politics, sport, tech) from 2004 - 2005. We've put a subset of these articles into a public [Google Cloud Storage](https://cloud.google.com/storage/) (GCS) bucket. Each of the articles is in a `.txt` file. -To examine the data and send it to the NL API, we'll write a Python script to read each text file from GCS, send it to the `classifyText` endpoint, and store the results in a [BigQuery](https://cloud.google.com/bigquery/) table. BigQuery is Google Cloud's big data warehouse tool - it lets us easily store and analyze large datasets. +To examine the data and send it to the NL API, we'll write a Python script to read each text file from GCS, send it to the `classifyText` endpoint, and store the results in a [BigQuery](https://cloud.google.com/bigquery/) table. BigQuery is Google Cloud's big data warehouse tool - it lets us easily store and analyze large datasets. To see the type of text we'll be working with, run the following command to view one article (`gsutil` provides a command line interface for GCS): @@ -151,7 +154,7 @@ Before we send the text to the Natural Language API, we need a place to store th ![Navigate to the BigQuery web UI](https://storage.googleapis.com/aju-dev-demos-codelabs/images/bigquery1.png) -Then click on the dropdown arrow next to your project name and select __Create new dataset__: +Then click on the dropdown arrow next to your project name and select __Create new dataset__: ![Create a new BigQuery dataset](https://storage.googleapis.com/aju-dev-demos-codelabs/images/bigquery2.png) @@ -188,7 +191,7 @@ gcloud iam service-accounts keys create key.json --iam-account=my-account@$PROJE export GOOGLE_APPLICATION_CREDENTIALS=key.json ``` -Now we're ready to send the text data to the NL API. To do that we'll use a Python script using the Python module for Google Cloud (note that you could accomplish the same thing from many other languages; there are many different cloud [client libraries](https://cloud.google.com/apis/docs/cloud-client-libraries)). +Now we're ready to send the text data to the NL API. To do that we'll use a Python script using the Python module for Google Cloud (note that you could accomplish the same thing from many other languages; there are many different cloud [client libraries](https://cloud.google.com/apis/docs/cloud-client-libraries)). Bring up the `classify-text.py` file `walkthrough editor-open-file "code-snippets/ml/cloud_shell_tutorials/cloud-nl-text-classification/classify-text.py" "in the text editor"`, and in the code, **replace `YOUR_PROJECT`** with the name of your project. @@ -244,9 +247,9 @@ We're ready to start classifying articles and importing them to BigQuery. Run th python classify-text.py ``` -The script takes about two minutes to complete, so while it's running we'll discuss what's happening. +The script takes about two minutes to complete, so while it's running we'll discuss what's happening. -We're using the `google-cloud` [Python client library](https://googlecloudplatform.github.io/google-cloud-python/) to access Google Cloud Storage, the NL API, and BigQuery. First we create a client for each service we'll be using, and then we create references to our BigQuery table. +We're using the `google-cloud` [Python client library](https://googlecloudplatform.github.io/google-cloud-python/) to access Google Cloud Storage, the NL API, and BigQuery. First we create a client for each service we'll be using, and then we create references to our BigQuery table. `files` is a reference to each of the BBC dataset files in the public bucket. We iterate through these files, download the articles as strings, and send each one to the NL API's in our `classify_text` function. For all articles where the NL API returns a category, we save the article and its category data to a `rows_for_bq` list. When we're done classifying each article, we insert our data into BigQuery using `create_rows()`. @@ -261,7 +264,7 @@ Enter the following query in the **Compose Query** box, **first replacing `YOUR_ SELECT * FROM `YOUR_PROJECT.news_classification.article_data` ``` -You should see your data when the query completes. The `category` column has the name of the first category the NL API returned for our article, and `confidence` is a value between 0 and 1 indicating how confident the API is that it categorized the article correctly. +You should see your data when the query completes. The `category` column has the name of the first category the NL API returned for our article, and `confidence` is a value between 0 and 1 indicating how confident the API is that it categorized the article correctly. We'll learn how to perform more complex queries on the data in the next step. @@ -273,14 +276,14 @@ First, let's see which categories were most common in our dataset. Enter the fol ```sql #standardSQL -SELECT - category, - COUNT(*) c -FROM - `YOUR_PROJECT.news_classification.article_data` -GROUP BY - category -ORDER BY +SELECT + category, + COUNT(*) c +FROM + `YOUR_PROJECT.news_classification.article_data` +GROUP BY + category +ORDER BY c DESC ``` @@ -300,10 +303,10 @@ Or, we could get only the articles where the NL API returned a confidence score ```sql #standardSQL -SELECT - article_text, - category -FROM `YOUR_PROJECT.news_classification.article_data` +SELECT + article_text, + category +FROM `YOUR_PROJECT.news_classification.article_data` WHERE cast(confidence as float64) > 0.9 ``` diff --git a/ml/cloud_shell_tutorials/cloud-speech-intro/request.json b/ml/cloud_shell_tutorials/cloud-speech-intro/request.json new file mode 100644 index 0000000..da3d640 --- /dev/null +++ b/ml/cloud_shell_tutorials/cloud-speech-intro/request.json @@ -0,0 +1,10 @@ +{ + "config": { + "encoding":"FLAC", + "sample_rate": 16000, + "language_code": "en-US" + }, + "audio": { + "uri":"gs://cloud-samples-tests/speech/brooklyn.flac" + } +} diff --git a/ml/cloud_shell_tutorials/cloud-speech-intro/tutorial.md b/ml/cloud_shell_tutorials/cloud-speech-intro/tutorial.md index ba8b8e4..3d94a10 100644 --- a/ml/cloud_shell_tutorials/cloud-speech-intro/tutorial.md +++ b/ml/cloud_shell_tutorials/cloud-speech-intro/tutorial.md @@ -1,70 +1,92 @@ # Speech to Text Transcription with the Cloud Speech API -[Codelab Feedback](https://github.com/googlecodelabs/feedback/issues/new?title=[cloud-speech-intro]:&labels[]=content-platform&labels[]=cloud) - - ## Overview -*Duration is 1 min* - - The Cloud Speech API lets you do speech to text transcription from audio files in over 80 languages. -In this lab, we will record an audio file and send it to the Cloud Speech API for transcription. +In this lab, you will see how to send an audio to the Cloud Speech API for transcription. #### What you'll learn * Creating a Speech API request and calling the API with curl * Calling the Speech API with audio files in different languages -#### What you'll need +![Speech API logo](https://storage.googleapis.com/aju-dev-demos-codelabs/images/speech_api_sm.png) -* A Google Cloud Platform Project -* A Browser, such [Chrome](https://www.google.com/chrome/browser/desktop/) or [Firefox](https://www.mozilla.org/firefox/) +**Time to complete**: About 20 minutes +Click the **Continue** button to move to the next step. -## Create an API Key +## Create a Google Cloud Platform (GCP) project if you don't have one -*Duration is 2 min* +**If you already have a Google Cloud Platform project, you can skip this step**. +If you don't have a Google Cloud Platform (GCP) project yet, create one [here](https://cloud.google.com/free/). Be sure to sign up for free trial credits. +**Note the name of your new project** — you'll use that in the next step. -Since we'll be using curl to send a request to the Speech API, we'll need to generate an API key to pass in our request URL. To create an API key, navigate to the API Manager section of your project dashboard: +Return to this tab once you're done. + +## Set your project in the Cloud Shell and create an API Key + +First, run the following command to ensure that the Cloud Shell is using the correct GCP project +(replacing `` with the name of your project): + +```bash + gcloud config set project +``` -![8cbae8dc9ba56e1e.png](img/8cbae8dc9ba56e1e.png) +Next, since we'll be using curl to send a request to the Natural Language API, we'll need to generate an API key to pass in our request URL. -Then, navigate to the __Credentials__ tab and click __Create credentials__: +> **Note**: If you've already created an API key in this project during one of the other Cloud Shell tutorials, you can just use the existing key⸺you don't need to create another one. Just be sure to set the `API_KEY` environment variable with your existing key as described below. -![fc9b83db953a127a.png](img/fc9b83db953a127a.png) +To create an API key, navigate to: -In the drop down menu, select __API key__: +**APIs & services > Credentials** in the [Cloud Console](https://console.cloud.google.com/): -![bc4940935c1bef7f.png](img/bc4940935c1bef7f.png) +![apis_and_services](https://storage.googleapis.com/aju-dev-demos-codelabs/images/apis_and_services.png) -Next, copy the key you just generated. +Then click __Create credentials__: + +![create_credentials1](https://storage.googleapis.com/aju-dev-demos-codelabs/images/create_credentials1.png) + +In the drop-down menu, select __API key__: + +![create_credentials2](https://storage.googleapis.com/aju-dev-demos-codelabs/images/create_credentials2.png) + +Next, copy the key you just generated. Click __Close__. Now that you have an API key, save it to an environment variable to avoid having to insert the value of your API key in each request. You can do this in Cloud Shell. Be sure to replace `` with the key you just copied. -``` +```bash export API_KEY= ``` +Next, you'll enable the Speech API for your project, if you've not already done so. -## Create your Speech API request +## Enable the Speech API -*Duration is 2 min* +Click on [this link](https://console.cloud.google.com/flows/enableapi?apiid=speech.googleapis.com) to enable the Speech API for your project. +After you've enabled it, you don't need to do any further setup, as you've already set up an API key. Just return to this tab. -You can build your request to the speech API in a `request.json` file. First create this file in Cloud Shell: +Next, you'll use the Speech API to make a transcription request. -``` -touch request.json +## Create your Speech API request + +First, change to this directory in the cloud shell: + +```bash +cd ~/code-snippets/ml/cloud_shell_tutorials/cloud-nl-intro ``` -Open it using your preferred command line editor (`nano`, `vim`, `emacs`). Add the following to your `request.json` file, replacing the `uri` value with the uri of your raw audio file: +You'll remain in this directory for the rest of the tutorial. -#### __request.json__ +Bring up the `request.json` file +`walkthrough editor-open-file "code-snippets/ml/cloud_shell_tutorials/cloud-speech-intro/request.json" "in the text editor"`. -``` +It should look like this: + +```json { "config": { "encoding":"FLAC", @@ -84,19 +106,19 @@ In the `audio` object, you pass the API the uri of our audio file in Cloud Stora ## Call the Speech API -*Duration is 1 min* - -You can now pass your request body, along with the API key environment variable you saved earlier, to the Speech API with the following `curl` command (all in one single command line): +You can now pass your request body, along with the API key environment variable you saved earlier, to the Speech API with the following `curl` command: -``` +```bash curl -s -X POST -H "Content-Type: application/json" --data-binary @request.json \ "https://speech.googleapis.com/v1beta1/speech:syncrecognize?key=${API_KEY}" ``` +Notice that the curl command used the API key that you generated. + Your response should look something like the following: -``` +```json { "results": [ { @@ -118,16 +140,12 @@ You'll notice that we called the `syncrecognize` method in our request above. Th ## Speech to text transcription in different languages -*Duration is 2 min* - - -Are you multilingual? The Speech API supports speech to text transcription in over 80 languages! You can change the `language_code` parameter in `request.json`. You can find a list of supported languages [here](https://cloud.google.com/speech/docs/languages). +Are you multilingual? The Speech API supports speech to text transcription in over 80 languages! You can change the `language_code` parameter in your json request. You can find a list of supported languages [here](https://cloud.google.com/speech/docs/languages). -For example, if you had a Spanish audio file, you can set the `language_code` attributes in the `request.json` file like this: +For example, if you had a Spanish audio file, you could set the `language_code` attributes in the `request.json` file like this: -#### __request.json__ -``` +```json { "config": { "encoding":"FLAC", @@ -143,20 +161,30 @@ For example, if you had a Spanish audio file, you can set the `language_code` at ## Congratulations! - +`walkthrough conclusion-trophy` You've learned how to perform speech to text transcription with the Speech API. In this example you passed the API the Google Cloud Storage URI of your audio file. Alternatively, you can pass a base64 encoded string of your audio content. -#### __What we've covered__ +#### What we've covered * Passing the Speech API a Google Cloud Storage URI of an audio file * Creating a Speech API request and calling the API with curl * Calling the Speech API with audio files in different languages -#### Next Steps +#### Some next Steps -* Check out the Speech API [tutorials](https://cloud.google.com/speech/docs/tutorials) in the documentations. +* Check out the Speech API [tutorials](https://cloud.google.com/speech/docs/tutorials) in the documentation. * Try out the [Vision API](https://cloud.google.com/vision/) and [Natural Language API](https://cloud.google.com/natural-language/)! - +--------------- +Copyright 2018 Google Inc. All Rights Reserved. Licensed under the Apache +License, Version 2.0 (the "License"); you may not use this file except in +compliance with the License. You may obtain a copy of the License at +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +License for the specific language governing permissions and limitations under +the License. diff --git a/ml/cloud_shell_tutorials/cloud-translation-intro/tutorial.md b/ml/cloud_shell_tutorials/cloud-translation-intro/tutorial.md index 4e5ef70..8c7adbe 100644 --- a/ml/cloud_shell_tutorials/cloud-translation-intro/tutorial.md +++ b/ml/cloud_shell_tutorials/cloud-translation-intro/tutorial.md @@ -3,7 +3,7 @@ ## Overview -The [[Cloud Translation API](https://cloud.google.com/translate/) lets you translate an arbitrary string into any supported language using state-of-the-art Neural Machine Translation. +The [[Cloud Translation API](https://cloud.google.com/translate/) lets you translate an arbitrary string into any supported language using state-of-the-art Neural Machine Translation. In this lab, we'll use the Cloud Translation API from the command line to translate text and detect the language of text if the language is unknown. @@ -24,7 +24,8 @@ Return to this tab once you're done. ## Set your project in the Cloud Shell and create an API Key -First, run the following command to ensure that the Cloud Shell is using the correct GCP project— whether new or existing— as follows (replacing `` with the name of your project): +First, run the following command to ensure that the Cloud Shell is using the correct GCP project +(replacing `` with the name of your project): ```bash gcloud config set project @@ -33,7 +34,7 @@ First, run the following command to ensure that the Cloud Shell is using the cor Next, since we'll be using curl to send a request to the Translation API, we'll need to generate an API key to pass in our request URL. -> **Note**: If you've already created an API key in this project during one of the other Cloud Shell tutorials, you can just use the existing key⸺you don't need to create another one. +> **Note**: If you've already created an API key in this project during one of the other Cloud Shell tutorials, you can just use the existing key⸺you don't need to create another one. Just be sure to set the `API_KEY` environment variable with your existing key as described below. To create an API key, navigate to: @@ -61,7 +62,9 @@ Next, you'll enable the Translation API for your project, if you've not already ## Enable the Translation API -Click on [this link](https://console.cloud.google.com/flows/enableapi?apiid=translate.googleapis.com) to enable the Translation API for your project. (After you've enabled it, you don't need to do any further setup, as you've already set up an API key above.) +Click on [this link](https://console.cloud.google.com/flows/enableapi?apiid=translate.googleapis.com) to enable the Translation API for your project. + +After you've enabled it, you don't need to do any further setup, as you've already set up an API key. Just return to this tab. Next, you'll translate some text from English to Spanish. @@ -94,7 +97,7 @@ In the response, you can see that the translated text as well as the source lang ## Detect Language -In addition to translating text, the Translation API also lets you detect the language of text. In this example you will detect the language of two strings. Pass the text to be examined, along with the API key environment variable you saved earlier, to the Translation API with the following curl command: +In addition to translating text, the Translation API also lets you detect the language of text. In this example you will detect the language of two strings. Pass the text to be examined, along with the API key environment variable you saved earlier, to the Translation API with the following curl command: ```bash TEXT_ONE="Meu%20nome%20é%20Steven" @@ -134,7 +137,7 @@ The languages returned by this sample are "pt" and "ja". These are the [ISO-639 `walkthrough conclusion-trophy` -You've learned how to translate text with the Cloud Translation API! +You've learned how to translate text with the Cloud Translation API! #### What we've covered diff --git a/ml/cloud_shell_tutorials/cloud-vision-nl-translate/tutorial.md b/ml/cloud_shell_tutorials/cloud-vision-nl-translate/tutorial.md index 07a4bee..dd63453 100644 --- a/ml/cloud_shell_tutorials/cloud-vision-nl-translate/tutorial.md +++ b/ml/cloud_shell_tutorials/cloud-vision-nl-translate/tutorial.md @@ -29,7 +29,8 @@ Return to this tab once you're done. ## Set your project in the Cloud Shell and create an API Key -First, run the following command to ensure that the Cloud Shell is using the correct GCP project— whether new or existing— as follows (replacing `` with the name of your project): +First, run the following command to ensure that the Cloud Shell is using the correct GCP project +(replacing `` with the name of your project): ```bash gcloud config set project @@ -37,7 +38,7 @@ First, run the following command to ensure that the Cloud Shell is using the cor Next, since we'll be using curl to send a request to the Vision API, we'll need to generate an API key to pass in our request URL. -> **Note**: If you've already created an API key in this project during one of the other Cloud Shell tutorials, you can just use the existing key⸺you don't need to create another one. +> **Note**: If you've already created an API key in this project during one of the other Cloud Shell tutorials, you can just use the existing key⸺you don't need to create another one. Just be sure to set the `API_KEY` environment variable with your existing key as described below. To create an API key, navigate to: @@ -66,7 +67,8 @@ Next, you'll enable the Vision, Translate, and Natural Language APIs for your pr ## Enable the Vision, Translate, and Natural Language APIs Click on [this link](https://console.cloud.google.com/flows/enableapi?apiid=vision.googleapis.com,translate.googleapis.com,language.googleapis.com) to enable the Vision, Translate, and Natural Language APIs for your project. -(After you've enabled them, you don't need to do any further setup, as you've already set up an API key above.) + +After you've enabled them, you don't need to do any further setup, as you've already set up an API key. Just return to this tab. Next, you'll send a request to the Cloud Vision API.