Semantic Labeling

[sriramk117]

Description

Overview

This PR implements an ADA feeding perception node that enables semantic labeling. Semantic labeling acts as infrastructure in the perception code for advances to the user interface including: (1) allowing natural language prompting for ADA users, (2) auto-feeding features that give the robot the ability to continue to feed a specific food item on the plate until it is no longer there without user prompting, and (3) taking user preference (i.e. how they want to eat their meal) into account during bite selection.

ROS2 Interfaces

Firstly, the node creates an action server (GenerateCaption) that takes in a list of string labels describing the food items on the plate (for example, ['meat', 'cucumber']) and then, runs GPT-4o to compile these string labels into a visually-descriptive, sentence query for GroundingDINO (for example, "Food items including strips of grilled meat and seasoned cucumber spears arranged on a light gray plate.").

The node also creates an action server (SegmentAllItems) that runs a pipeline consisting of a vision language model called GroundingDINO and the segmentation model EfficientSAM/SAM to output masks paired with semantic labels describing the masks for each of the detected food items on the plate.

Implementation Details

The perception node uses three foundation models to achieve semantic labeling: GPT-4o, GroundingDINO, and EfficientSAM/SegmentAnything.

From experimentation with prompting techniques on GroundingDINO and GroundingDINO's feature extraction functionality, we determined that prompting GroundingDINO with a sentence prompt that contains descriptive qualifiers results in significantly better performance for bounding box detection. Therefore, we query GPT-4o with the latest above plate image and a list of strings by the user labeling the food items on the plate to generate a visually-descriptive sentence incorporating those labels to describe the image. This sentence then gets used as input for GroundingDINO.

See the current system and user queries for GPT-4o below:

System Query:

You are a prompt engineer that is assigned to describe items  
in an image you have been queried so that a vision language model 
can take in your prompt as a query and use it to for classification 
tasks. You respond in string format and do not provide any explanation 
for your responses.  

User Query:

Your objective is to generate a sentence prompt that describes the food
items on a plate in an image. 
You are given an image of a plate with food items and a list of the food items 
on the plate.
Please compile the inputs from the list into a sentence prompt that effectively
lists the food items on the plate.
Add qualifiers to the prompt to better visually describe the food for the VLM
to identify. Don't add any irrelevant qualifiers.

Here is the input list of food items to compile into a string: {labels_list}

Here are some sample responses that convey how you should format your responses:

Food items including grapes, strawberries, blueberries, melon chunks, and 
carrots on a small, blue plate.

Food items including strips of grilled meat and seasoned cucumber 
spears arranged on a light gray plate.

Food items including baked chicken pieces, black olives, bell pepper slices, 
and artichoke on a plate.

GroundingDINO is a vision language model that detects objects in an image (through bounding boxes) given natural language prompts. We use GroundingDINO as it pairs semantic labels from the natural language prompt with each of the detected bounding boxes.

Then, we use the segmentation model EfficientSAM/SAM to extract pixel-wise masks of each of the detected objects given their bounding boxes.

Here, we share a couple images of GroundingDINO's bounding box detection performance on various above plate image photos:

Set Up Steps

Follow these steps to install necessary dependencies. :

• Open a new terminal window and go into your ADA workspace.
• Go into the ada_feeding directory: cd src/ada_feeding
• Run pip install -r requirements.txt

Testing Procedure

Test Setup

Before running any isolated tests, the following list of commands should be run each in separate terminal windows. Also, ensure that source install/setup.bash gets called in each of the windows that will run ROS commands.

• Launch the web app:
  • First, open three separate terminal tabs/windows and go into the correct folder in each window: cd src/feeding_web_interface/feedingwebapp
  • Run the following listed commands in the separate terminal tabs/windows:
    • Launch the web app server: npm run start
    • Set the web rtc server: node --env-file=.env server.js
    • Start the robot browser: node start_robot_browser.js
• Launch the perception nodes: ros2 launch ada_feeding_perception ada_feeding_perception.launch.py
• Launch the dummy nodes: ros2 launch feeding_web_app_ros2_test feeding_web_app_dummy_nodes_launch.xml run_food_detection:=false run_face_detection:=false
• Launch the nano bridge sender: ros2 launch nano_bridge sender.launch.xml
• Launch the nano bridge receiver: ros2 launch nano_bridge receiver.launch.xml

Below, see isolated tests for each of the two action servers that enable semantic labeling (the SegmentAllItems action and the GenerateCaption action).

Test GPT-4o Action:

• Open a separate terminal window. To send action goals to the GenerateCaption action server, use this command: ros2 action send_goal /GenerateCaption ada_feeding_msgs/action/GenerateCaption "input_labels: ['placeholder']"
  • In place of the placeholder, you would have a list of strings that concisely label the different food items to be eaten during the meal. Here is an example send goal ROS command you could use for this action: ros2 action send_goal /GenerateCaption ada_feeding_msgs/action/GenerateCaption "input_labels: ['blueberries', 'carrots', 'strawberries', 'green grapes']"
• If the action succeeds, you will find the result returned as a string called caption.

Test GroundingDINO + SegmentAnything vision pipeline:

• Open a separate terminal window. To send action goals to the SegmentAllItems action server, use this command: ros2 action send_goal /SegmentAllItems ada_feeding_msgs/action/SegmentAllItems "caption: Some input sentence query for the pipeline that visually describes the food items on the plate."
  • Here is an example of a sample action goal you could send which includes an example input caption: ros2 action send_goal /SegmentAllItems ada_feeding_msgs/action/SegmentAllItems "caption: Food items on a blue plate including blueberries, carrots, strawberries, and green grapes."
• If the action succeeds, it will return a list of Mask messages (containing information like the region of interest and a binary mask of the segmented item) named detected_items and a list of strings named item_labels
  • Each semantic label of the item_labels list corresponds to the Mask message at the same index of the detected_items list

I believe the most useful way to test the performance of semantic labeling is by merging these isolated tests together. First, send a goal to the GenerateCaption action server and get the resulting caption. Then send this caption as input to the SegmentAllItems action. This will be more identical to how the web interface sends goals to each of the action servers and fetches results to display during the bite selection stage.

Testing Semantic Labeling on Different Images

An important part of testing is experimenting with various above plate images that have different food items and analyzing the performance of these perception nodes on these different images. To change the image that is used on the feeding web app dummy nodes (and hence, the one that gets passed as input into each of the action servers), follow these steps:

• Open the file feeding_web_app_ros2_test/launch/feeding_web_app_dummy_nodes_launch.xml in your favorite IDE:
• Pick an rgb image and its corresponding depth image from the folder feeding_web_app_ros2_test/data
• Change the image file names in the rgb_path and depth_path launch arguments accordingly
• Then follow the isolated tests detailed above to test semantic labeling on these images

Before opening a pull request

• Format your code using black formatter python3 -m black .
• Run your code through pylint and address all warnings/errors. The only warnings that are acceptable to not address is TODOs that should be addressed in a future PR. From the top-level ada_feeding directory, run: pylint --recursive=y --rcfile=.pylintrc ..

Before Merging

• Squash & Merge

[jjaime2]

Looking very good overall. Could likely use some cleanup but the core functionality seems to make sense. I would also check for pylint warnings if you haven't since I noticed many macros in the old code that this PR was based off of.

[jjaime2]

I don't see a .env file added in this PR, but I'm guessing this was more for personal use. I'd recommend omitting this change unless it's relevant for the functionality of the PR.

[jjaime2]

I noticed more references to a env file later in the code, where exactly does this come into play?

[sriramk117]

I'm adding environment variable functionality to our codebase so we can privately store API keys without exposing them publicly in github. In this particular case, it is for accessing the PRL OpenAI API key to invoke GPT-4o.

[sriramk117]

I'm assuming this may come in handy later on as well if we power perception w/ foundation models in the future.

[jjaime2]

Comment seems misleading, I suspect this was an old comment for item_labels

[jjaime2]

Related to another comment on .gitignore. Are we using .env files for something?

[jjaime2]

Is this meant to be logged?

[amalnanavati]

Also, why are you multiplying by 1000?

In general, I find it more readable to specify the units of measurements after the variable name itself, e.g., in this case maybe it should be inference_time_ms given the factor of 1000?

[jjaime2]

For this and other instances of bbox, does it have any properties like xmin, xmax, ymin, ymax to make this easily understandable?

[amalnanavati]

You coudl consider using a namedtuple for this

[jjaime2]

Old code for SegmentFromPoint didn't require a type-cast to float, it's fine if this is needed just wanted to check since I noticed the discrepancy.

[amalnanavati]

Also, these small changes are precisely why its best to not copy-paste code (e.g., if casting to float improves the code here, we should ideally percolate it back to SegmentFromPoint). So this is one more of the functions that should be in a helper.

Another thing you can consider is making SegmentAllItems inherit from the SegmentFromPoint class, and only override some functions.

[jjaime2]

Should mention the goal_handle here

[jjaime2]

Also known as "Goal succeeded", also typo in "VIsion"

[amalnanavati]

Nit: add newlines at the end of files. (I know not all files have it, but in general it is a best practice so we should enforce it on new/modified files)

[amalnanavati]

Nit: newline.

I think your empty line at the end of the file has some whitespae, which is why Github doesn't recognize it as the empty line at the end of the file.

[amalnanavati]

Nit: does this belong in model or config? This seems more like configuration. I believe what is downloaded to model is the actual files storing model weights.

[amalnanavati]

Also, some of these, like time and random, are "standaard imports." pylint should help take care of some of these issues. Eventually, I want to set up precommit on this repo so reformatting and some level of linting happens automatically

[amalnanavati]

I'd suggest having a parameter to the function (set by default to 1) that controls how many of the top masks you use per food item

[amalnanavati]

Same comment as above re. ms vs sec and adding units to variable names

[amalnanavati]

I recently learned that rates live forever in a node unless explicitly deleted by destroy_rate (see comment here). So I'd recommend adding that at the end of this function, else the rate will keep taking up callback resources even after the action has finished.

[amalnanavati]

Are there cases where the vision pipeline could hang? If so, I'd recommend adding a timeout to the action (maybe in the action message itself) to be robust to that

[amalnanavati]

Why not do this in the cleanup function itself? (both here and below)