How to obtain and interpret explanations of predictions
BigQuery ML is an easy-to-use way to invoke machine learning models on structured data using just SQL. Although it started with only linear regression, more sophisticated models like Deep Neural Networks and AutoML Tables have been added by connecting BigQuery ML with TensorFlow and Vertex AI as its backend. …
Create an ARIMA model, then detect anomalies
Let’s say we have a time series dataset of the number of bicycles rented in London every day:
SELECT
EXTRACT(date from start_date) AS start_date,
COUNT(*) AS num_trips
FROM `bigquery-public-data.london_bicycles.cycle_hire`
GROUP BY start_dateHow can we find unusual days in terms of the number of bicycle rentals?
In BigQuery, all we…
BigQuery ML can use Vertex AI to tune common model parameters
What is BigQuery ML?
BigQuery ML allows you to quickly train ML models on data in BigQuery. For example, suppose you want to train a linear ML model to predict the duration of a bicycle rental, you can do that using:
CREATE OR REPLACE MODEL ch09eu.bicycle_model_linear
OPTIONS(
model_type='linear_reg', input_label_cols=['duration']
)
AS
SELECT
start_station_name,
CAST(EXTRACT(DAYOFWEEK FROM…This tip is short enough to be a Twitter thread. I’m preserving it here:
A neat trick that uses Stored Procedure with a BigQuery script
My colleague Polong Lin did a brilliant thing recently. He wanted to do a demo, and rather than ask us to follow a checklist of instructions, he simply had us make a single BigQuery call:
CALL
`ai-analytics-solutions`.publicdata.setup_flights_demo(
'MY-PROJECT-NAME', 'demods')This call provisioned everything in our BigQuery project that we needed…
Why do we need it, how good is the code-free ML training, really, and what does all this mean for data science jobs?
The Google Cloud AI Platform team have been heads down the past few months building a unified view of the machine learning landscape. This was launched today at Google I/O as Vertex AI. What is it? How good is it? What does it mean for data science jobs?
What is there to unify?
The idea…
Converting rows to columns
Sometimes, you might want to reformat a table result so you have separate columns for each unique value. This is called a Pivot table — normally, it’s only a display function supported by BI tools. However, it can occasionally be helpful to create the pivot table in SQL. …
Using the new geospatial capabilities in Data Studio
Let’s take the new geospatial capabilities in Data Studio for a spin by exploring the public dataset of NOAA Storm Prediction Center storm reports of hail. Please follow along with me — there’s a free BigQuery sandbox that you can use. Data Studio is a free product.
Querying in BigQuery
Just turn it on; no code changes needed
copy of article published in https://cloud.google.com/blog/topics/developers-practitioners/speeding-small-queries-bigquery-bi-engine on 2021–04–01.
A quick and easy way to speed up small queries in BigQuery (such as to populate interactive applications or dashboards) is to use BI Engine. …
Using Cloud Dataflow and Google Cloud Public Datasets
Originally posted on Google Cloud Blog at https://cloud.google.com/blog/products/gcp/how-to-do-distributed-processing-of-landsat-data-in-python
One common data analysis task across the agricultural industry, as well in academia and government (for drought studies, climate modeling, and so on), is to create a monthly vegetation index from Landsat images, which is now available as a public dataset on…
