mirror/gradio
2
0
Fork 0
mirror of https://github.com/gradio-app/gradio.git synced 2025-01-18 10:44:33 +08:00
Code Issues Packages Projects Releases Wiki Activity

Improve flagging guide & add flagging with Blocks (#1853)

Browse Source 
* improving guide

* flagging guide

* remove img

* fixed spaces

* fixes
This commit is contained in:
Abubakar Abid 2022-07-22 12:14:34 -07:00 committed by GitHub
parent 9360670e23
commit c638aa9961
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 90 additions and 26 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
demo/blocks_flag/requirements.txt Normal file
View File

@ -0,0 +1 @@
numpy

33
demo/blocks_flag/run.py Normal file
View File

@ -0,0 +1,33 @@
import numpy as np
import gradio as gr
def sepia(input_img, strength):
sepia_filter = strength * np.array(
[[0.393, 0.769, 0.189], [0.349, 0.686, 0.168], [0.272, 0.534, 0.131]]
) + (1-strength) * np.identity(3)
sepia_img = input_img.dot(sepia_filter.T)
sepia_img /= sepia_img.max()
return sepia_img
callback = gr.CSVLogger()
with gr.Blocks() as demo:
with gr.Row():
with gr.Column():
img_input = gr.Image()
strength = gr.Slider(0, 1, 0.5)
img_output = gr.Image()
with gr.Row():
btn = gr.Button("Flag")
# This needs to be called at some point prior to the first call to callback.flag()
callback.setup([img_input, strength, img_output], "flagged_data_points")
img_input.change(sepia, [img_input, strength], img_output)
strength.change(sepia, [img_input, strength], img_output)
# We can choose which components to flag -- in this case, we'll flag all of them
btn.click(lambda *args: callback.flag(args), [img_input, strength, img_output], None, _preprocess=False)
if __name__ == "__main__":
demo.launch()

82
guides/using_flagging.md
View File

@ -1,36 +1,34 @@
# Using Flagging
Related spaces: https://huggingface.co/spaces/aliabd/calculator-flagging-crowdsourced, https://huggingface.co/spaces/aliabd/calculator-flagging-options, https://huggingface.co/spaces/aliabd/calculator-flag-basic
Related spaces: https://huggingface.co/spaces/gradio/calculator-flagging-crowdsourced, https://huggingface.co/spaces/gradio/calculator-flagging-options, https://huggingface.co/spaces/gradio/calculator-flag-basic
Tags: FLAGGING, DATA
## Introduction
When you deploy or demo a machine learning model, you may find that it behaves differently than how you expected (e.g. the model makes an incorrect prediction) when a user tries it with their own data. Capturing these "hard" data points is important because it allows you to make you machine learning model more reliable and robust.
When you demo a machine learning model, you might want to collect data from users who try the model, particularly data points in which the model is not behaving as expected. Capturing these "hard" data points is valuable because it allows you to improve your machine learning model and make it more reliable and robust.
Gradio simplifies the collection of this data by including a FLAG button with every `Interface`. This allows your user or tester to easily send data back to you, whether the model is running locally or has been shared by setting `share=True`.
Gradio simplifies the collection of this data by including a **Flag** button with every `Interface`. This allows a user or tester to easily send data back to the machine where the demo is running. In this Guide, we discuss more about how to use the flagging feature, both with `gradio.Interface` as well as with `gradio.Blocks`.
## The **Flag** button
## The **Flag** button in `gradio.Interface`
Underneath the output interfaces, there is a button marked **Flag**. When a user testing your model sees input with interesting output, such as erroneous or unexpected model behaviour, they can flag the input for the interface creator to review.
Flagging with Gradio's `Interface` is especially easy. By default, underneath the output components, there is a button marked **Flag**. When a user testing your model sees input with interesting output, they can click the flag button to send the input and output data back to the machine where the demo is running. The sample is saved to a CSV log file (by default). If the demo involves images, audio, video, or other types of files, these are saved separately in a parallel directory and the paths to these files are saved in the CSV file.
![flag button](/assets/guides/flag_button.gif)
There are [four parameters](/docs/#interface-header) in `gradio.Interface` that control how flagging works. We will go over them in greater detail.
There are four parameters `gr.Interface` that control how flagging works. We will go over them in greater detail.
* `allow_flagging`: this parameter can be set to either `"manual"`, `"auto"`, or `"never"`.
* `manual`: users will see a button to flag, and events are only flagged when it's clicked.
* `auto`: users will not see a button to flag, but every event will be flagged automatically.
* `never`: users will not see a button to flag, and no event will be flagged.
* `allow_flagging`: this parameter can be set to either `"manual"` (default), `"auto"`, or `"never"`.
* `manual`: users will see a button to flag, and samples are only flagged when the button is clicked.
* `auto`: users will not see a button to flag, but every sample will be flagged automatically.
* `never`: users will not see a button to flag, and no sample will be flagged.
* `flagging_options`: this parameter can be either `None` (default) or a list of strings.
* If `None`, then the user simply clicks on the **Flag** button and no additional options are shown.
* If a list of strings are provided, this allows user to select from a list of options when flagging. Only applies if `allow_flagging` is `"manual"`.
* The chosen option is then piped along with the input and output.
* If a list of strings are provided, then the user sees several buttons, corresponding to each of the strings that are provided. For example, if the value of this parameter is `["Incorrect", "Ambiguous"]`, then buttons labeled **Flag as Incorrect** and **Flag as Ambiguous** appear. This only applies if `allow_flagging` is `"manual"`.
* The chosen option is then logged along with the input and output.
* `flagging_dir`: this parameter takes a string.
* It represents what to name the directory where flagged data is stored.
* `flagging_callback`: this parameter takes an instance of a subclass of the `FlaggingCallback` class
* Using this parameter allows you to write custom code that gets run when the flag button is clicked
* By default, this is set to an instance of `gr.CSVLogger`
* One example is setting it to an instance of `gr.HuggingFaceDatasetSaver` which can allow you to pipe any flagged data into a HuggingFace Dataset.
* One example is setting it to an instance of `gr.HuggingFaceDatasetSaver` which can allow you to pipe any flagged data into a HuggingFace Dataset. (See more below.)
## What happens to flagged data?
@ -63,8 +61,7 @@ iface = gr.Interface(
iface.launch()
```
<iframe src="https://hf.space/embed/aliabd/calculator-flag-basic/+" frameBorder="0" height="500" title="Gradio app" class="container p-0 flex-grow space-iframe" allow="accelerometer; ambient-light-sensor; autoplay; battery; camera; document-domain; encrypted-media; fullscreen; geolocation; gyroscope; layout-animations; legacy-image-formats; magnetometer; microphone; midi; oversized-images; payment; picture-in-picture; publickey-credentials-get; sync-xhr; usb; vr ; wake-lock; xr-spatial-tracking" sandbox="allow-forms allow-modals allow-popups allow-popups-to-escape-sandbox allow-same-origin allow-scripts allow-downloads"></iframe>
<gradio-app space="gradio/calculator-flag-basic/"></gradio-app>
When you click the flag button above, the directory where the interface was launched will include a new flagged subfolder, with a csv file inside it. This csv file includes all the data that was flagged.
@ -98,7 +95,7 @@ im/0.png,Output/0.png,2022-02-04 19:49:58.026963
im/1.png,Output/1.png,2022-02-02 10:40:51.093412
```
If you wish for the user to provide a reason for flagging, you can pass a list of strings to the `flagging_options` argument of Interface. Users will have to select one of the strings when flagging, which will be saved as an additional column to the CSV.
If you wish for the user to provide a reason for flagging, you can pass a list of strings to the `flagging_options` argument of Interface. Users will have to select one of these choices when flagging, and the option will be saved as an additional column to the CSV.
If we go back to the calculator example, the following code will create the interface embedded below it.
```python
@ -112,7 +109,7 @@ iface = gr.Interface(
iface.launch()
```
<iframe src="https://hf.space/embed/aliabd/calculator-flagging-options/+" frameBorder="0" height="500" title="Gradio app" class="container p-0 flex-grow space-iframe" allow="accelerometer; ambient-light-sensor; autoplay; battery; camera; document-domain; encrypted-media; fullscreen; geolocation; gyroscope; layout-animations; legacy-image-formats; magnetometer; microphone; midi; oversized-images; payment; picture-in-picture; publickey-credentials-get; sync-xhr; usb; vr ; wake-lock; xr-spatial-tracking" sandbox="allow-forms allow-modals allow-popups allow-popups-to-escape-sandbox allow-same-origin allow-scripts allow-downloads"></iframe>
<gradio-app space="gradio/calculator-flagging-options/"></gradio-app>
When users click the flag button, the csv file will now include a column indicating the selected option.
@ -123,11 +120,17 @@ num1,operation,num2,Output,flag,timestamp
6,subtract,1.5,3.5,off by one,2022-02-04 11:42:32.062512
```
## Doing more with the data
## The HuggingFaceDatasetSaver Callback
Suppose you want to take some action on the flagged data, instead of just saving it. Perhaps you want to trigger your model to retrain, or even just share it with others in a cloud dataset. We've made this super easy with the `flagging_callback` parameter.
Sometimes, saving the data to a local CSV file doesn't make sense. For example, on Hugging Face
Spaces, developers typically don't have access to the underlying ephemeral machine hosting the Gradio
demo. That's why, by default, flagging is turned off in Hugging Face Space. However,
you may want to do something else with the flagged data.
We've made this super easy with the `flagging_callback` parameter.
For example, below we're going to pipe flagged data from our calculator example into a Hugging Face Dataset, e.g. so that we can build a "crowd-sourced" dataset:
For example, below we're going to pipe flagged data from our calculator example into a crowd-sourced Hugging Face Dataset.
```python
import os
@ -139,6 +142,7 @@ iface = gr.Interface(
calculator,
["number", gr.Radio(["add", "subtract", "multiply", "divide"]), "number"],
"number",
description="Check out the crowd-sourced dataset at: [https://huggingface.co/datasets/aliabd/crowdsourced-calculator-demo](https://huggingface.co/datasets/aliabd/crowdsourced-calculator-demo)",
allow_flagging="manual",
flagging_options=["wrong sign", "off by one", "other"],
flagging_callback=hf_writer
@ -146,16 +150,42 @@ iface = gr.Interface(
iface.launch()
```
<iframe src="https://hf.space/embed/aliabd/calculator-flagging-crowdsourced/+" frameBorder="0" height="500" title="Gradio app" class="container p-0 flex-grow space-iframe" allow="accelerometer; ambient-light-sensor; autoplay; battery; camera; document-domain; encrypted-media; fullscreen; geolocation; gyroscope; layout-animations; legacy-image-formats; magnetometer; microphone; midi; oversized-images; payment; picture-in-picture; publickey-credentials-get; sync-xhr; usb; vr ; wake-lock; xr-spatial-tracking" sandbox="allow-forms allow-modals allow-popups allow-popups-to-escape-sandbox allow-same-origin allow-scripts allow-downloads"></iframe>
You can now see all the examples flagged above in this [public HF dataset](https://huggingface.co/datasets/aliabd/crowdsourced-calculator-demo/blob/main/data.csv).
Notice that we define our own
instance of `gradio.HuggingFaceDatasetSaver` using our Hugging Face token and
the name of a dataset we'd like to save samples to. In addition, we also set `allow_flagging="manual"`
because on Hugging Face Spaces, `allow_flagging` is set to `"never"` by default. Here's our demo:
<gradio-app space="gradio/calculator-flagging-crowdsourced/"></gradio-app>
You can now see all the examples flagged above in this [public Hugging Face dataset](https://huggingface.co/datasets/aliabd/crowdsourced-calculator-demo).
![flagging callback hf](/assets/guides/flagging-callback-hf.png)
We created the `gr.HuggingFaceDatasetSaver` class, but you can pass your own custom class as long as it inherits from `FLaggingCallback` defined in [this file](https://github.com/gradio-app/gradio/blob/master/gradio/flagging.py). If you create a cool callback, please contribute it to the repo!
We created the `gradio.HuggingFaceDatasetSaver` class, but you can pass your own custom class as long as it inherits from `FLaggingCallback` defined in [this file](https://github.com/gradio-app/gradio/blob/master/gradio/flagging.py). If you create a cool callback, contribute it to the repo!
## Flagging with Blocks
What about if you are using `gradio.Blocks`? On one hand, you have even more flexibility
with Blocks -- you can write whatever Python code you want to run when a button is clicked,
and assign that using the built-in events in Blocks.
At the same time, you might want to use an existing `FlaggingCallback` to avoid writing extra code.
This requires two steps:
1. You have to run your callback's `.setup()` somewhere in the code prior to the
first time you flag data
2. When the flagging button is clicked, then you trigger the callback's `.flag()` method,
making sure to collect the arguments correctly and disabling the typical preprocessing.
Here is an example with an image sepia filter Blocks demo that lets you flag
data using the default `CSVLogger`:
$code_blocks_flag
$demo_blocks_flag
## Privacy
Please make sure your users understand when the data they submit is being saved, and what you plan on doing with it. This is especially important when you use `allow_flagging=auto`. We suggest including this info in the description so that it's read before the interface.
Important Note: please make sure your users understand when the data they submit is being saved, and what you plan on doing with it. This is especially important when you use `allow_flagging=auto` (when all of the data submitted through the demo is being flagged)
### That's all! Happy building :)